Connecting to your cluster using Dex

In a previous article, we explored how to connect to an SKS cluster with OIDC using GitLab as the OpenID provider. In this article we’ll introduce Dex, a CNCF Sandbox project, which adds an abstraction layer in front of the identity provider. We’ll demonstrate how to use Dex to connect to an SKS cluster with user and group management handled within GitLab.

About Dex

As defined in the official documentation: “Dex is an identity service that uses OpenID Connect to drive authentication for other apps. Dex acts as a portal to other identity providers through connectors.” This lets Dex defer authentication to LDAP servers, SAML providers, or Identity Providers (IdP) like GitHub and Google.”.

Dex is handy for abstracting the potential complexity or translating the specific protocol of the underlying IdP. For example, when using Google as an IdP, group information is not included in the OIDC token. This requires an additional API call to retrieve this information, which Dex handles in the background. Additionally, Dex allows users to be configured in its local database, enabling authentication even if the IdP is temporarily unavailable. Dex is a great fit for large organizations which integrate with multiple IdPs or require custom authentication logic. For smaller setups with simpler needs, a direct connection to the IdP may be sufficient.

The schema below illustrates Dex’s overall architecture. If an application uses OIDC with Dex, each authentication request initiated by a user is sent to Dex. Dex then redirects the request to the configured identity provider via a connector. Once the authentication is done, the identity provider sends the OIDC token back to Dex, which returns it to the application.

When used in the context of Kubernetes, the client application is the kubectl binary. The following sequence diagram illustrates the interactions between the various actors involved. The oidc-login plugin, a widely used kubectl plugin in charge of communicating with OIDC providers (Dex in our example), will be discussed later.

Next, we’ll configure our GitLab environment.

Creating a GitLab application

To use GitLab as the identity provider, we need to create an application. In this article, we use Techwhale, a GitLab group created for demo purposes.

From the Settings/ Applications menu of this GitLab group, we create an application named K8S-DEX-OIDC. We set the redirect URL to https://auth.techwhale.io/callback and select the following scopes:

• read_user
• openid
• profile
• email

Once the application is created, it provides the client ID and Secret which we save in the GITLAB_CLIENT_ID and GITLAB_CLIENT_SECRET environment variables. We’ll use these values in a later section.

Creating a cluster specifying OIDC parameters

When creating an SKS cluster, specific parameters can be used to configure the OIDC connectivity.

• client-id: OpenID client ID
• issuer-url: OpenID provider URL
• username-claim: JWT claim to use as the user’s name
• username-prefix: Prefix prepended to username claims
• groups-claim: JWT claim to use as the user’s group
• groups-prefix: Prefix prepended to group claims
• required-claim: a map that describes a required claim in the ID Token

The cluster creation process was covered in the previous article. In this example, it’s important to set these parameters with the following values:

• client-id: “kubectl”
• issuer-url: “https://auth.techwhale.io”
• groups-claim: “groups”
• groups-prefix: “oidc:”
• username-claim: “preferred_username”
• username-prefix: “oidc:”

This configuration enables authentication via Dex, which will be available on https://auth.techwhale.io.

Installing infrastructure components

In this section, we’ll install the following components in the cluster:

• Traefik Ingress Controller to expose applications outside the cluster
• Cert-Manager to automatically issue and renew TLS certificates
• Dex acting as our identity provider abstraction layer

Traefik

Traefik is a widely used Ingress Controller in Kubernetes. We’re installing it using Helm:

helm install traefik oci://ghcr.io/traefik/helm/traefik -n traefik --create-namespace

Next, we get the IP address of the Load balancer exposing Traefik:

$ kubectl get svc -n traefik
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
my-traefik   LoadBalancer   10.102.52.194   XXX.XXX.XXX.XXX   80:30653/TCP,443:30783/TCP   75s

We use this IP address to create an A record, in our DNS provider, for the auth.techwhale.io subdomain.

Cert manager:

Cert-manager automates the certificate issuance and renewal process. We’re installing Cert-manager using Helm:

helm repo add cert-manager https://charts.jetstack.io
helm install cert-manager cert-manager/cert-manager --set crds.enabled=true --version 1.16.2 -n cert-manager --create-namespace

Next, we create a ClusterIssuer responsible for issuing certificates through Let’s Encrypt Certification Authority:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: acme
spec:
 acme:
 email: devops@techwhale.io
 server: https://acme-v02.api.letsencrypt.org/directory
 privateKeySecretRef:
 name: acme-account-key
 solvers:
 - http01:
 ingress:
 class: traefik
EOF

Installing DEX

Before installing Dex, we make sure the following environment variables are set:

• GITLAB_CLIENT_ID: identifier of the GitLab application created previously
• GITLAB_CLIENT_SECRET: secret associated with this same application
• KUBECTL_CLIENT_ID: we can set the value to “kubectl” as this is Dex’s client application
• KUBECTL_CLIENT_SECRET: we can use the value “StaticClientSuperSafeSecret” for this example

Since we’ll be installing Dex using Helm, we begin by creating the values.yaml configuration file as follows:

cat <<EOF > values.yaml
config:
 issuer: https://auth.techwhale.io

 storage:
 type: memory

 connectors:
 - type: gitlab
 id: gitlab
 name: GitLab
 config:
 baseURL: https://gitlab.com
 clientID: ${GITLAB_CLIENT_ID}
 clientSecret: ${GITLAB_CLIENT_SECRET}
 redirectURI: https://auth.techwhale.io/callback
 useLoginAsID: false
 scopes: 
 - openid
 - profile
 - email
 - groups

 staticClients:
 - id: ${KUBECTL_CLIENT_ID}
 redirectURIs:
 - "http://localhost:8000"
 name: 'Kubectl client'
 secret: ${KUBECTL_CLIENT_SECRET}

ingress:
 enabled: true
 className: traefik
 annotations:
 cert-manager.io/cluster-issuer: acme
 hosts:
 - host: auth.techwhale.io
 paths:
 - path: /
 pathType: ImplementationSpecific
 tls:
 - hosts:
 - auth.techwhale.io
 secretName: certs
EOF

This file defines the configuration of Dex’s connector, GitLab in this example. It also defines the kubectl client, within the staticClients property, which will interact with Dex.

We’re installing Dex using Helm:

helm repo add dex https://charts.dexidp.io
helm upgrade --install dex dex/dex --version 0.19.1 -f values.yaml -n dex --create-namespace

Updating the kubeconfig file

We’ll be using kubelogin (a.k.a kube-login), a kubectl plugin that allows kubectl to communicate with an OIDC identity provider. In the previous article, we used GitLab as the identity provider, in the current example we’re still using GitLab, but we’ll be abstracting this identity provider through Dex. When making a kubectl call, the user will be redirected to GitLab, via Dex, for authentication.

Once the plugin is installed, we can verify the content of the ID Token with the following command (which also helps in case we need to troubleshoot the configuration):

kubectl oidc-login setup --oidc-issuer-url=https://auth.techwhale.io --oidc-client-id=${KUBECTL_CLIENT_ID} --oidc-client-secret=${KUBECTL_CLIENT_SECRET} --oidc-extra-scope=groups --oidc-extra-scope=profile

This command returns the oidc-login configuration information, which we use to define a new user in the kubeconfig.

kubectl config set-credentials oidc \
 --exec-api-version=client.authentication.k8s.io/v1beta1 \
 --exec-command=kubectl \
 --exec-arg=oidc-login \
 --exec-arg=get-token \
 --exec-arg=--oidc-issuer-url=https://auth.techwhale.io \
 --exec-arg=--oidc-client-id=${KUBECTL_CLIENT_ID} \
 --exec-arg=--oidc-client-secret=${KUBECTL_CLIENT_SECRET} \
 --exec-arg=--oidc-extra-scope=groups \
 --exec-arg=--oidc-extra-scope=profile

Before using this new user, we’ll create the appropriate roles in the cluster.

GitLab groups

Several subgroups and users were created in TechWhale GitLab group:

• user techwhale-dev is member of subgroup techwhale/viewer
• user techwhale-admin is member of subgroup techwhale/admin

Next, we’ll use Kubernetes RBAC to restrict these subgroups’ access to a specific list of actions and resource types.

ClusterRole / ClusterRoleBinding based on GitLab’s groups

Our requirements are as follows:

• Users in the techwhale/viewer group should only have read-only access to the main resources in the cluster (Deployments, Pods, ConfigMaps, …)
• Users in the techwhale/admin group should have full access to these resources

To meet these requirements we create the necessary ClusterRoles and ClusterRoleBindings. These resources define the authorizations for both GitLab groups.

cat <<EOF | kubectl apply -f -
# Admin ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: gitlab-oidc-admin
rules:
- apiGroups: [""]
 resources: ["pods", "services", "configmaps", "secrets"]
 verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: ["apps"]
 resources: ["deployments", "daemonsets", "statefulsets"]
 verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: ["batch"]
 resources: ["jobs", "cronjobs"]
 verbs: ["get", "list", "watch", "create", "update", "delete"]
---
# Admin ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: gitlab-oidc-admin
subjects:
- kind: Group
 name: "oidc:techwhale/admin"
 apiGroup: rbac.authorization.k8s.io
roleRef:
 kind: ClusterRole
 name: gitlab-oidc-admin
 apiGroup: rbac.authorization.k8s.io
---
# Viewer ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: gitlab-oidc-viewer
rules:
- apiGroups: [""]
 resources: ["pods", "services", "configmaps"]
 verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
 resources: ["deployments", "daemonsets", "statefulsets"]
 verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
 resources: ["jobs", "cronjobs"]
 verbs: ["get", "list", "watch"]
---
# Viewer ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: gitlab-oidc-viewer
subjects:
- kind: Group
 name: "oidc:techwhale/viewer"
 apiGroup: rbac.authorization.k8s.io
roleRef:
 kind: ClusterRole
 name: gitlab-oidc-viewer
 apiGroup: rbac.authorization.k8s.io
EOF

Now that everything is set up, we’ll verify the authentication is working as expected.

Verification

First, we set the oidc user in the kubeconfig as the default. This ensures the kubectl command will be handled by Dex unless another user is specified:

kubectl config set-context --current --user=oidc

Next, we ensure we are signed out from gitlab.com in our browser. Then we create a simple Deployment:

kubectl create deploy www --image=nginx:1.24

This command opens a browser to GitLab and requests authentication. First, we log in as techwhale-admin

Next, we accept the permissions requested by the GitLab application providing access to our account:

Then the authentication is validated.

In the terminal, we observe that the Deployment was correctly created.

deployment.apps/www created

We log out of GitLab. Next, we run the same command once more.

kubectl create deploy www --image=nginx:1.24

We authenticate with user techwhale-dev.

Then, we accept the requested permissions as we did previously.

In the terminal, we observe that the Deployment cannot be created because the user does not have the required permissions.

error: failed to create deployment: deployments.apps is forbidden: User "oidc:techwhale-dev" cannot create resource "deployments" in API group "apps" in the namespace "default"

Using GitLab groups we are thus able to limit the access to our cluster’s resources.

Key takeaways

Companies often rely on an SSO (Single Sign-On) solution to manage authentication and authorization across the entire organization. Configuring Kubernetes to use these solutions is straightforward, as a cluster has OIDC parameters designed for this purpose. Dex takes this integration a step further acting as an abstraction layer in front of identity providers. It provides a single interface for multiple OIDC providers, with a connector for each, including GitLab, Google, and LDAP, simplifying client configuration.