Secure Docker Image Pulls from Cloudsmith to Kubernetes using OIDC

Pulling Docker images from private registries for containerised applications presents a security challenge. It requires authentication management, network access, and trust across distributed systems. Credentials must be securely handled and rotated, and image pulls can break due to network restrictions or expired tokens. All of this makes deployment and security harder.

To address this, you can use OpenID Connect (OIDC) when pulling Docker images from Cloudsmith into Kubernetes. OIDC adds a layer to OAuth 2.0 that identifies who is making the request. Cloudsmith can then verify that the requests come from a trusted source and can grant short-lived access tokens. So you no longer have to rely on long-lived credentials, enhancing security and simplifying credential management.

The Challenge: Secure Image Pulling

Kubernetes clusters often need to pull Docker images from private registries like Cloudsmith. Traditionally, this meant storing long-lived credentials in the cluster, which has some security risks:

1. Credentials could be compromised if the cluster is breached.
2. Regular rotation of credentials becomes a manual, error-prone process.
3. It's hard to apply the principle of least privilege effectively.

Understanding Image Pull Secrets

Before we dive into the solution, let's review what an Image Pull Secret is in Kubernetes. 

An Image Pull Secret is a way to pass credentials to the kubelet to authenticate with a private container registry. Typically, the secret contains:

1. The registry URL
2. Username
3. Password or access token

While Image Pull Secrets solves the immediate authentication problem, using them with long-lived credentials still exposes us to the security risks mentioned earlier. The credentials don’t automatically expire, leaving them vulnerable to misuse if leaked or unrotated.

The OIDC Solution

OpenID Connect (OIDC) allows us to use short-lived tokens instead of long-lived credentials, reducing risk exposure. Here's how it works in the context of Kubernetes and Cloudsmith:

Kubernetes has its own identity in the form of a service account. The service account has an associated JWT (JSON Web Token). During runtime, the following steps are taken:

1. Kubernetes issues the signed OIDC token (a JWT).
2. Cloudsmith receives and verifies the JWT and exchanges it for a short-lived Cloudsmith token.
3. Kubernetes uses the short-lived Cloudsmith token to create or update an Image Pull Secret.
4. At runtime, the pod requests to pull an image.
5. Kubernetes pulls an image from Cloudsmith using the Image Pull Secret for authentication.
6. Kubelet deploys the image to the pod.

Automating the Process with a CronJob

To automate the process, we use a Kubernetes CronJob. This job runs at regular intervals and performs the following tasks:

1. Retrieves the Kubernetes service account token.
2. Exchange service account token for a Cloudsmith token via OIDC.
3. Create or update an Image Pull Secret with the new Cloudsmith token.

Here's an example of what this CronJob might look like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloudsmith-secret-creator
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: secret-manager
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "update", "patch", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-manager-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: cloudsmith-secret-creator
  namespace: default
roleRef:
  kind: Role
  name: secret-manager
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cloudsmith-secret-creator
  namespace: default
spec:
  schedule: "0 */12 * * *"
  concurrencyPolicy: Replace
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cloudsmith-secret-creator
          volumes:
          - name: shared-data
            emptyDir: {}
          initContainers:
          - name: fetch-and-parse-token
            image: nicolaka/netshoot
            env:
            - name: CLOUDSMITH_ORG
              value: "iduffy-demo"  # Replace with your Cloudsmith org name
            - name: CLOUDSMITH_SERVICE_SLUG
              value: "kubernetes"  # Replace with your Cloudsmith service slug
            - name: CLOUDSMITH_DOCKER_SERVER
              value: "docker.cloudsmith.io"  # Replace if necessary
            volumeMounts:
            - name: shared-data
              mountPath: /shared
            command:
            - /bin/sh
            - -c
            - |
              set -e
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              curl -sSL -X POST -H "Content-Type: application/json" \
                -d "{\"oidc_token\":\"$TOKEN\", \"service_slug\":\"$CLOUDSMITH_SERVICE_SLUG\"}" \
                https://api.cloudsmith.io/openid/$CLOUDSMITH_ORG/ | \
                jq -r '.token' > /shared/cloudsmith_token.txt
              echo $CLOUDSMITH_SERVICE_SLUG > /shared/service_slug.txt
              echo $CLOUDSMITH_DOCKER_SERVER > /shared/docker_server.txt
          containers:
          - name: create-secret
            image: bitnami/kubectl:latest
            volumeMounts:
            - name: shared-data
              mountPath: /shared
            command:
            - /bin/sh
            - -c
            - |
              set -e
              CLOUDSMITH_TOKEN=$(cat /shared/cloudsmith_token.txt)
              CLOUDSMITH_SERVICE_SLUG=$(cat /shared/service_slug.txt)
              CLOUDSMITH_DOCKER_SERVER=$(cat /shared/docker_server.txt)
              kubectl create secret docker-registry cloudsmith-registry-secret \
                --docker-server="$CLOUDSMITH_DOCKER_SERVER" \
                --docker-username="$CLOUDSMITH_SERVICE_SLUG" \
                --docker-password="$CLOUDSMITH_TOKEN" \
                --dry-run=client -o yaml | kubectl apply -f -
          restartPolicy: OnFailure

This CronJob: 

1. Runs every 12 hours

2. Gets the Kubernetes service account token.

3. Exchanges it for a Cloudsmith token.

4. Creates or updates an Image Pull Secret named cloudsmith-pull-secret.

Benefits of the OIDC Approach

Implementing this OIDC-based solution offers significant benefits:

1. Automatic Rotation: The CronJob ensures the token is regularly refreshed, maintaining a good security posture without manual intervention.
2. Simplified Management: There's no need to manually update credentials, reducing operational overhead.
3. Principle of Least Privilege: The token only has the permissions it needs for a limited time, aligning with security best practices.

Using the Image Pull Secret

Once the Image Pull Secret is created or updated by the CronJob, you can use it in your pod specifications like this:

apiVersion: v1
kind: Pod
metadata:
  name: cloudsmith-demo-pod
  namespace: default
spec:
  containers:
  - name: demo-container
    image: docker.cloudsmith.io/CLOUDSMITH_ORG/CLOUDSMITH_REPO/ubuntu:latest
    command: ["sleep", "3600"]  # Sleep for 1 hour, just as an example
  imagePullSecrets:
  - name: cloudsmith-registry-secret

This pod will use the cloudsmith-pull-secret to authenticate with Cloudsmith and pull the specified image.

Conclusion

Using OIDC to pull Docker images from Cloudsmith into Kubernetes eliminates the need for long-lived credentials. It reduces the risk of leaked credentials, removes the need for manual credential rotation, and aligns with current security best practices.

Authentication in Kubernetes is evolving with significant recent enhancements. Kubernetes v1.20 saw the introduction of credential provider plugins. These allow the kubelet to dynamically request credentials for a container registry instead of storing static credentials on disk. This was enhanced in v1.33 to support automatic use of the service account token for image pulls. This enables short-lived authentication without needing to manage Image Pull Secrets and reduces complexity. Cloudsmith remains vigilant, and these updates help with our goal of increasing automation in supply chain security.

Liked this article? Don\'t be selfish (:-), share with others:   Tweet