Introduction​

The world of Kubernetes has evolved rapidly, transforming the way organizations manage containerized applications. In this age of dynamic workloads and cloud-native architecture, security, and access control are paramount. Argo CD and Paralus are two powerful tools that have emerged to address these concerns, enabling organizations to implement zero-trust principles in their Kubernetes environments. In this blog, we'll explore the concepts behind Argo CD and Paralus and delve into their integration for zero-trust Kubernetes access.

Understanding Argo CD​

Argo CD, an open-source GitOps continuous delivery tool, simplifies the deployment and management of applications in Kubernetes clusters. GitOps is a methodology that uses Git as a single source of truth for both infrastructure and application configurations. Argo CD watches your Git repository for changes and ensures that the actual state of the cluster matches the desired state defined in the repository.

Key features of Argo CD include

• Declarative Application Configuration: Applications are defined in Git repositories using YAML files, making it easy to version and manage configurations.
• Automated Synchronization: Argo CD continuously monitors Git repositories for changes, ensuring that the cluster is always in the desired state.
• Role-Based Access Control (RBAC): Argo CD allows you to define granular access control policies for users and teams, enhancing security.

Paralus for Zero Trust Kubernetes Access​

Paralus is a powerful tool designed to provide zero-trust access to Kubernetes clusters. Zero trust, as a security paradigm, assumes that threats can come from both inside and outside the network, and no entity should be implicitly trusted. Paralus integrates with Kubernetes to ensure that every interaction is authenticated and authorized, irrespective of the user's location.

Key features of Paralus include

• Authentication: Users must authenticate themselves before accessing the Kubernetes cluster. Paralus supports various authentication methods, such as username/password, single sign-on using openid connect (sso), and more.
• Role-Based Access Control: Access to Kubernetes resources is determined by policies and roles, ensuring only authorized users can perform specific actions.
• Audit Trails: Every action taken in the Kubernetes cluster is logged, providing transparency and accountability.

Use Case: Secure Access to Kubernetes with Argo CD and Paralus​

Let's dive into a practical use case to understand how Argo CD and Paralus work together to provide Zero Trust Kubernetes access.

Scenario​

Imagine a mid-sized tech company with multiple Kubernetes clusters deployed in various cloud providers and on-premises data centers. The company uses Argo CD for continuous deployment of applications and wants to strengthen the security posture of their clusters.

Using Argo CD to manage Paralus applications​

We can leverage Argo CD to deploy and manage Paralus core controller apps and relay agents across these multiple kubernetes clusters

Paralus Core: Install and manage paralus app as an argocd helm application

In this example we are going to deploy paralus core to the same cluster as argocd, you can choose a different destination cluster. This would make paralus upgrades automatic on every release depending on your synchronization option and leverage the power for argocd.

argocd app create paralus-core

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: paralus-core
spec:
  destination:
    name: ''
    namespace: paralus
    server: 'https://kubernetes.default.svc'
  source:
    path: charts/ztka
    repoURL: 'https://github.com/paralus/helm-charts.git'
    targetRevision: HEAD
  helm:
    parameters:
      - name: deploy.postgresql.enable
        value: 'true'
      - name: kratos.kratos.development
        value: 'true'
    sources: []
  project: default
  syncPolicy: null
  syncOptions:
    - CreateNamespace=true

Next up synchronize the application to deploy Paralus core components to your target cluster

Now you can access paralus console after configuring your DNS Settings

Relay Agents: All the Paralus imported clusters can be configured as separate argocd applications. This enables managing and upgrading relay agents by just updating the version in git repo. Very useful to upgrade a fleet of agents across imported clusters.

argocd app create gke-dev

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gke-dev
spec:
  destination:
    name: ''
    namespace: paralus-system
    server: 'https://35.224.124.44'
  source:
    path: argocd/cloud/gke/dev/
    repoURL: 'https://github.com/niravparikh05/paralus-integrations.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Integrating Argo CD with Paralus​

The integration of Argo CD with Paralus enhances Kubernetes access control and security, promoting a zero-trust approach. Here's how it works

• Authentication: When a user attempts to access the Kubernetes cluster through Argo CD, Paralus handles the authentication process. Users must provide their credentials and complete any additional authentication steps, to verify their identity.
• Access Control: Paralus enforces access policies based on the user's identity and role. Even within Argo CD, users can only interact with resources and applications they are authorized to access.
• Auditability: Paralus logs all interactions within the Kubernetes cluster, providing a detailed audit trail. This ensures that all actions are recorded and can be reviewed for security and compliance purposes.

This is possible by adding clusters in argocd using the kubeconfig generated using paralus, in this example we will add a cluster with kubeconfig generated having service account with full cluster access

argocd cluster add gke-dev --name gke-dev-paralus --kubeconfig ./kubeconfig-admin@paralus.local.yaml

WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `gke-dev` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0003] ServiceAccount "argocd-manager" already exists in namespace "kube-system" 
INFO[0004] ClusterRole "argocd-manager-role" updated    
INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" updated 
Cluster 'https://94a82cca-76ce-4cbb-8a36-8acc25eca1bd.user.ic-oss.dev.rafay-edge.net:443' added

Now deploying any application to above destination cluster will adhere to paralus’ just in time access and can be tracked via paralus' audit logs

Let’s say we add a new guestbook application to above cluster as our destination

argocd app create guestbook

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
spec:
  destination:
    name: ''
    namespace: default
    server: >-
      https://94a82cca-76ce-4cbb-8a36-8acc25eca1bd.user.ic-oss.dev.rafay-edge.net:443
  source:
    path: guestbook
    repoURL: 'https://github.com/argoproj/argocd-example-apps.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Next up synchronize the guestbook app

Verifying the auditlogs, we can see that the guestbook application has been deployed with paralus system user having necessary permissions.

By integrating Argo CD with Paralus, organizations can achieve a robust, zero-trust security model that safeguards their Kubernetes infrastructure and the applications running within it.

Conclusion​

In the ever-evolving landscape of Kubernetes, ensuring security and access control is of utmost importance. Argo CD and Paralus represent two powerful tools that, when combined, can provide a robust zero-trust approach to Kubernetes access.

By using Argo CD for GitOps-driven application deployment and Paralus for strong authentication, role-based access control, and auditability, organizations can confidently embrace the world of container orchestration without compromising security. This integration not only secures the Kubernetes environment but also enhances compliance and accountability.

In summary, Argo CD and Paralus are a dynamic duo for implementing a zero-trust approach to Kubernetes access, providing a secure and reliable foundation for modern containerized applications.

In late Sep 2023, we had the opportunity to speak, present and participate at DevOpsCon 2023 in New York City. In this blog, we will briefly describe what we presented at the conference and our observations about the event itself.

DevOpsCon is a global conference focused on CI/CD, Kubernetes Ecosystem, Agile & Lean Business. If you live outside the United States, you may want to attend one of their conferences in other cities such as Munich, Singapore, London and Berlin.

This year's conference in New York was held at the Marriott near Brooklyn Bridge which is a fantastic location, literally right across from the Brooklyn Bridge and NYU Engineering.

Our goals at this conference were very simple

• Educate the attendees about various approaches for secure access to Kubernetes clusters and their pros/cons.

Our Presentation​

Secure developer and operator access to Kubernetes clusters is a big deal for organizations. Paralus, an open source project has been a major contributor of innovative solutions to address this challenge.

Teams that ignore this issue or handle this poorly not only have to deal with major security and compliance issues, but also land up significantly impacting their application developer velocity.

Here is the title of our presentation and we also had a live demonstration to help the attendees understand how this worked behind the scenes.

Learnings​

We also had the opportunity to meet and speak with many of the attendees who were at the conference. Most of the attendees were from the greater NY metro area and some were from Boston and the DC area. There were three common themes that bubbled up constantly.

Kubernetes - Part of a larger infrastructure​

Most had Kubernetes resident applications leveraging managed services (e.g. RDS managed database etc.) in cloud providers. They said they were struggling with creation and ongoing management of these complex environments.

Acute need for developer self-service​

Developers seem to be struggling with lack of self service for test beds and other environments. This said this was impacting their development velocity.

Generative AI​

Everyone seems to be talking about it. But, they are also struggling to get started and seem to be unsure where to apply it first in their applications. Most were of the opinion that the first and logical use case would that of a co-pilot.

Conclusion​

Organizations looking for a standalone, open source alternative can use Paralus. This is a CNCF project sponsored and maintained by the Rafay Team.

The event organizers will provide a live stream of my presentation and live demo. This should be available to everyone very soon.

Organizations today need to efficiently manage resources, ensure security, and foster innovation—all within shared environments. MultiTenancyCon, a co-located event at KubeCon is where the brightest minds in the industry come together to address these challenges head-on.

MultiTenancyCon​

Multi-TenancyCon is a vendor-neutral conference designed to foster collaboration, discussion, and knowledge sharing on Multi-Tenancy framework and technologies, and challenges for implementing and adopting it in a production set-up. It's a congregation of brilliant minds in the world of Kubernetes, focused on addressing the challenges and opportunities that multi-tenancy presents.

Why Multitenancy in Kubernetes?​

Multi-tenancy in Kubernetes refers to the practice of running multiple, isolated workloads on the same cluster, where each workload can belong to a different team, application, or customer. While the advantages of multi-tenancy are evident, such as resource optimization and cost efficiency, it's not without its complexities. Kubernetes provides the building blocks, but orchestrating them effectively requires expertise.

Current Challenges in Kubernetes Multitenancy​

With multi-tenant clusters, there are certain challenges that customers have to solve from a security and operational efficiency standpoint.

Some of those include:

• Access Control: Enabling a Zero Trust model for K8s infrastructure access
• Runtime security: Monitoring and detecting abnormal behavior and possible threats in real time
• Noisy Neighbor Issues: Enforcing resource quotas and ensuring isolation of namespaces belonging to different teams/applications
• Policy Enforcement: Implementing necessary guardrails to ensure compliance (e.g. appropriate labels for resources)
• Resource Utilization & Chargeback: Understanding usage and apportioning costs across teams/applications

Meet Team Paralus at MultiTenancyCon​

At this year's MultitenancyCon, Paralus maintainer Naveen Chakrapani and Falco maintainer Thomas Labarussias will be participating in a talk, Enhanced Security for Multi-Tenant Architectures: Key Components and Best Practices that will focus on access control and runtime security.

Access Control with Paralus​

Paralus is a powerful access management solution designed to simplify and centralize access control for Kubernetes. It enables a Zero Trust Access Model with Just-In-Time (JIT) account creation, automated RBAC (Role-Based Access Control) federation, and centralized audit trails for all user/kubectl activities.

Key benefits of Paralus include​

• Secure access management
• Automated RBAC
• Auditability

In addition to the talk, you can also meet team #paralus at KubeCon, kiosk F35 and learn how Paralus can help implement a Zero Trust Access model for K8s access and reduce the cognitive load for platform teams and developers.

Conclusion​

The challenges of multi-tenancy in Kubernetes are not to be underestimated. But, with the right knowledge and tools, this complexity can be tamed. MultitenancyCon is your opportunity to gain that knowledge, access those tools, and join a community of experts in the field.

If you're passionate about Kubernetes, multi-tenancy, and the future of cloud-native technologies, don't miss MultitenancyCon. It's your ticket to taming the complexity and unleashing the true potential of multi-tenancy in Kubernetes.

Organizations today need to efficiently manage resources, ensure security, and foster innovation—all within shared environments. MultiTenancyCon, a co-located event at KubeCon is where the brightest minds in the industry come together to address these challenges head-on.

Modern IT infrastructures are becoming more dynamic and resilient with the passing years. Containers, for example, have revolutionized the way IT organizations develop, deploy and manage applications.

While container management and orchestration seem to be the way to go nowadays, one of the key pain points is once we have the clusters and applications deployed, how do we guard our ever-growing ecosystem internally and externally?

Securing Kubernetes and cluster access management is complicated, but following the zero trust principles, you can ensure that only verified and authorized requests can have access to the information. As part of creating awareness around securing Kubernetes with zero trust principles, I recently gave a talk around this topic at Kubernetes Community Days Bengaluru 2023 that you can also watch. This blog post is based on my talk. Here, we will cover various aspects related to zero trust security and access management with Kubernetes with a practical application.

What is Zero Trust Security?​

Zero trust is a security framework that requires strict identity verification and authentication for every user and device attempting to access an organization's resources, regardless of whether they are inside or outside the network perimeter. Based on authentication, authorization, and encryption technologies, the purpose of zero trust is to continuously validate the security configurations of your infrastructure and applications to ensure trust across an environment. This approach improves protection against potential security breaches and unauthorized access.

Zero trust security incorporates these basic principles:

• Continually verify and validate the identity of each user connecting to your organization.
• Regularly verify and validate each asset connecting to your organization including networks, systems, applications, and devices.
• Monitor and control access to each user, asset, and resource, ensuring that they are provided the least privilege access to accomplish a task.
• Provide Just-in-Time (JIT) access. So, when a user or service needs access, they can have it and once the task is completed, the access would be revoked.
• Continuously track and log all operations done on the system through a comprehensive monitoring process.

Let’s learn a bit more about advanced zero trust security principles in detail.

Authentication​

Authentication is a process of verifying the identity of every process or registered users before enabling access to networks and systems. Kubernetes zero trust requires the authentication of every user and service account before enabling the execution of an API call.

This can be done using one of the following:

• Client certificates
• Bearer tokens
• OpenID Connect (OIDC) tokens
• HTTP basic authentication
• Authentication proxies
• Webhook token authentication

Authorization​

Authorization is the process of granting or denying access to specific resources, services or actions based on the identity and permissions of a user or a system entity.

Kubernetes uses a modular and extensible architecture for authorization, allowing cluster administrators to choose from various authorization mechanisms based on their specific requirements

• Node Authorization (Node): This authorization mode is used to control API access based on the node's identity. Nodes are granted specific permissions to interact with the Kubernetes API accordingly.
• Webhook Mode (Webhook): Webhook authorization allows Kubernetes to delegate authorization decisions to external systems through HTTP callbacks.
• Role Based Access Control (RBAC): This method provides access authorization based on the user’s role within the organization (i.e., developer, security, admin, etc.).

Admission Control​

Admission control is responsible for gatekeeping. Its dynamic nature validates if the request is allowed to be processed by the policies (can be business logic etc.).

Provisioning Just In Time Access​

It means provisioning access to resources for users upon request and revoking the access once the said task/session is completed. For more details, you can read this blog on understanding just-in-time (JIT) access in Paralus.

Auditing & Logging​

From a governance perspective, it is essential to keep a track of all the requests and their corresponding impact on the target system. This can be done by intercepting the Kubernetes API calls and logging them in your preferred database/logging system, inc.

How to implement Zero Trust Security using Paralus?​

Paralus is a CNCF Sandbox Project. It is an open source tool that enables controlled & audited access to Kubernetes infrastructure. It comes with JIT service account creation and user-level credential management that can integrate with your RBAC and SSO.

How does Paralus implement Zero Trust Security?​

Paralus uses Ory Kratos for identity management and authentication, and Casbin for authorization. As an additional security mechanism, Paralus implements mTLS across components where Paralus core acts as a certificate authority provider and validates all requests coming in from various services and apps. You can read the documentation to learn more about Paralus’ architecture.

Paralus acts as a proxy and intercepts all requests coming into the target Kubernetes clusters. Once the requests are authenticated and authorized, Paralus creates a service account with the necessary Kubernetes RBAC. It enables JIT access and also keeps a record of all incoming requests to achieve auditing across tens of clusters from a single place.

As part of KCD Bengaluru, I gave a demonstration on how Paralus can be installed and a few playbook scenarios that are common across organizations using Kubernetes. I also covered how Paralus can be used to your benefit in such cases.

Installing Paralus​

In this tutorial, we will install Paralus on a kind cluster. If you don’t already have one setup, you can follow this installation guide for kind and follow these steps for Paralus.

Once Paralus is installed, we can import existing clusters that can be accessed from Paralus.

Let’s see how Paralus implements mTLS across components. For every peer, user, relay-agent, a certificate signing request will be created which will be signed by Paralus certificate authority and used for verification on every request.

Log from relay-server

{"level": "info", "ts": "2023-06-26708:05:46.1382", "caller": "relay/relay.go:385", "msg": "Relay Server::peer certificate: -----BEGIN CERTIFICATE-----MIICETCCADagAWIBAgIRANQYS6T]DQON3VVC/CWCgYIKoZ1Z]-----END CERTIFICATE-----"}

{"level": "info", "ts" :"2023-66-26T08:05:46.1482", "caller": "relay/relay.go:497", "msg": "Relay Server: :user certificate: -----BEGIN CERTIFICATE-----MIICSJCCATGOAWIBAGIRAKOLUOBQAfZa/XVPLISHTNOWCGYIKOZIZJOEAWT-----END CERTIFICATE-----"}

{"level": "info", "ts": "2023-66-26T08:05:46.1552", "caller": "relay/relay.go: 594", "msg": "Relay Server: :connector certificate: -----BEGIN CERTIFICATE-----HIICQZCCAQGAWIBAQIQQ2UWLUV3G2LKTI1UO13g20AKBggqhkOPQ-----END CERTIFICATE-----"}

Log from relay-agent (core-connector certificate)

{"level": "info", "ts": "2023-06-26710:54:13.371Z", "caller": "agent/agent.go:407", "msg" : "Relay Agent::certificate: -----BEGIN CERTIFICATE-----MIICBTCCAaqgAWIBAQIRANTDYZVLAGEQh3mRUMDKS+kwCgYIKOZIZJOEAWIWFZEMM-----END CERTIFICATE-----"}

$ openssl x509 -in agent.crt -text -noout


Certificate:
Data:
Version: 3 (0x2)
serial Number
d4:db:63:3b:cb:76:01: :79:91:52:67:64:4bze9
signature Algorithm: ecdsa-with-SHA256
Issuer: C = USA, ST = California, L = Sunnyvale, 0 = Paralus, OU = Paralus Sentry, CN = paralus-core-relay

validity
Not Before Jun 26 10:54:13 2023 GMT
Not After Jun 23 10:54:13 2033 GMT

Subject: OU = chtf6gqc6pouns3fqcng, CN = cicmrq2c6pouns3fqdeg
Subject Public Key Info
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:

4:68:cd:72:ad:7e:43:3f1ab:3b:0
0:6C:93:2d:d6:5e:37:b0:74:82:10: a4: e3:bd:b2:
C:95:95
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extension:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Authority Key Identifier
key1d:06:C2:39:98:FF:B6:06:A1:

:69:98:FE:82:95:39:DB:A8:0B:E8

Signature Algorithm: ecdsa-with-SHA256
30:46:02:21:00: fc: f3:92:38:f3:7:64:44:5d:95:b9:02:ac:

Scenarios where Paralus is Helpful​

Let’s run through a few scenarios where Paralus proved to be beneficial.

Scenario 1: Someone new joins our organization as a developer. We would like to onboard them onto our AWS project and give them access to all AWS clusters.

Note: In this demo, we will use local users. You can also configure Paralus to use your organization’s identity provider as a single sign-on application for a more seamless experience.

Create a new group called “AWS Project Members”, which provides its members cluster admin access to all clusters under the AWS project. You can learn more about how to work with groups in Paralus.

Now, we will be creating a user in Paralus and making them a part of the “AWS Project Members” group.

That’s it. New users will now be able to access all clusters under the AWS project and execute kubectl commands against it.

As part of this scenario, we saw that a user once authenticated by the organization and based on the authorization policy, (in this case project roles associated with groups a user is a member of) is able to gain access to clusters and start executing kubectl commands.

Let’s verify just-in-time access provisioning. In your target cluster, you can check that a service account for the user just got created after the user executed their first command against it:

$ kubectl get sa -n paralus-system | grep “user”

virat-46-kohli-64-paralus-46local                        1                   5m55s

Scenario 2: Let’s say a trainee was accidentally given cluster admin access and is able to delete critical deployments running as part of his exploration. The project manager wants to revoke the access immediately and schedule a meeting to go over do’s and don'ts before assigning him read-only access.

Whoops, trainee Shivam deleted a critical Nginx deployment by mistake.

Virat, the project manager, wonders what happened and checks the audit logs.

Upon knowing it was Shivam who accidentally deleted it, Virat immediately decides to revoke his access and schedule a meeting on do’s and don'ts before adding him to the trainee group where he ideally should have been added.

Now Shivam is unable to access any of the clusters.

Let’s verify whether his access is actually removed from the target cluster.

$ kubectl get sa -n paralus-system

NAME                                   SECRETS                         AGE
admin-64paralus-46local                  1                             58m
default                                  1                             59m
system-sa                                1                             59m
virat-46kohli-64paralus-46local          1                             58m

Once added back to the trainee group, he’s able to view all the system resources for restricted namespaces and continue exploring without modifying/deleting anything.

Now, Shivam can view the resources but cannot modify/delete anything.

Conclusion​

Zero trust access is a crucial security framework in today's evolving threat landscape. It offers a more robust and proactive approach to cybersecurity, providing organizations with the ability to defend against a wide range of threats and protect sensitive data, no matter where their employees or resources are located. Embracing zero trust access principles is essential for staying ahead in the ongoing battle against cyber threats. In this article, we learned what zero trust security is, and a few basic principles. Further, we explored in depth the various zero trust principles like “authentication”, “authorization”, “just-in-time-access”, "admission control” and the importance of “auditing”. We also talked about how Paralus implements these various principles and also saw a demo of how Paralus can be installed and a few scenarios to verify the above in action.

I hope you found this post informative and engaging. For more posts like this one, do subscribe to our newsletter for a weekly dose of cloud native. I’d love to hear your thoughts on this post, let’s connect and start a conversation on Twitter or LinkedIn.

Introduction​

Managing Kubernetes clusters at scale can be a daunting task, especially when dealing with multiple clusters and the need to federate access and manage RBAC for a group of developers. However, the integration of two open-source projects, Kamaji and Paralus, provides an innovative solution to address these challenges. In this blog post, we will explore how Kamaji's Kubernetes-in-Kubernetes approach and Paralus' centralized RBAC and authentication features work together seamlessly to simplify cluster management, enhance security, and streamline RBAC operations.

With over 150 speakers and 20,000 attendees PlatformCon 2023, the virtual conference created by and for platform engineers was a hub of activity and excitement for tech enthusiasts and industry professionals alike. Team Paralus was also a part of such a vibrant community for the second edition of PlatformCon.

This event brought together a global community of techies, developers, and platform engineers to share the latest advancements and trends in the field of platform engineering.

The event also featured a diverse range of talks ranging from platform-culture, platform-blueprint, platform-impact to platform-stories. You can watch all the sessions on the Platform Engineering YouTube Channel.

Applications today have become advanced and complex. They can perform simple tasks like adding up two numbers to controlling a rocket in space. Irrespective of what the application does, ensuring that the application and the underlying infrastructure is safe and secured is critical.

So as teams and organizations build applications, we want to ensure that their infrastructure is secured. That was one of the reasons we came up with Paralus. We went a step further and donated it to the CNCF to help the community and also in turn build Paralus.

June 28, 2022, was an extremely special day for all of us here at Paralus. It was the day when Paralus was born. The day when we presented Paralus to the world. A lot of tools exist today for managing Kubernetes access. From the default role-based access control mechanism to custom DIY solutions along with a few open-source projects. However, most of these tools aren't easy to configure especially when it comes to individual clusters.

It’s a great day for all of us in the Paralus community as Paralus is now listed as a 1-click app on the DigitalOcean marketplace. You’ve seen in the past how easy it is to install Paralus on various environments, right from public cloud platforms like Amazon EKS, Microsoft AKS, Google GKE to your own laptop using Micro K8s and Kind.

However, with the launch of Paralus 1-click app on DigitalOcean marketplace, it’s even easier for you to try Paralus. In fact, we recommend using this 1-click offering to try Paralus quickly. So how does it work? Read on to learn more.