Simplifying Confidential Computing: Sealed Secrets in Action

The CNCF Confidential Containers (CoCo) project offers a feature called “sealed-secrets” that simplifies the process of deploying K8s workloads as confidential containers. A sealed secret is an encapsulated secret available only within a Trusted Execution Environment (TEE) after verifying the integrity of the TEE environment. Using a sealed secret is like using any other K8s secret.

If you are new to confidential containers (CoCo), I would recommend you to read my introductory blog on CoCo and also refer the materials in the CoCo project website.

There are two types of sealed secrets in CoCo: envelope and vault. In this blog, I’ll discuss the vault type sealed secret.

A vault type sealed secret is a pointer to a secret stored and managed remotely by ‘Trustee’.

Trustee is the name given to the components of the CNCF CoCo project that helps with attesting the TEE and providing secrets.

It includes the following key components:

• Key Broker Service (KBS): KBS is the remote attestation and secret delivery server. The TEEs talk to KBS for remote attestation and secret retrieval. KBS integrates with internal or external Key Management Systems (KMS) for secret storage and life-cycle management.
• Attestation Service (AS): AS verifies the TEE evidence.
• Reference Value Provider Service (RVPS): RVPS manages reference values to verify TEE evidence.

For more details on the Trustee components and overall attestation process, kindly refer to the following material.

When deploying your K8s workload in a TEE using CoCo, the CoCo runtime components that runs inside the TEE will retrieve the actual secret from Trustee and make it available to the workload. This secret retrieval process happens transparently.

At a high level, using sealed secrets in a CoCo workload involves the following steps:

1. Create a JSON formatted sealed secret config file with the location and other metadata about the actual secret.
2. Use the base64url encoded version of the sealed secret config file to create the sealed secret.
3. Create K8s secret from the sealed secret created in the previous step and use it in your workload.
4. The CoCo runtime will perform attestation with Trustee and unseal the secret inside the TEE for use by the workload.

The following diagram is a pictorial representation of the process:

Src: Author Created

Let’s see an end-to-end example of a vault type sealed secret.

The first step is to create a json file containing the location of the secret.

Below is an example JSON file, secret.json. Note the type, name, and the provider.

$ cat secret.json

{
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/mysecret/secret",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {}
}

The type of the sealed secret is vault and provider is kbs, indicating that the secret is managed by Trustee.

The name attribute specifies the location of the actual secret. It’s available via the KBS at default/mysecret/secret.

The sealed secret is of the form — sealed.JWS header.JWS body(sealed_secret_config_json).signature

For using sealed secrets in K8s, we use the following format:

sealed.fakejwsheader.base64url(sealed_secret_config_json).fakesignature

For our example, it’ll be

sealed.fakejwsheader.base64url(content_of_secret.json).fakesignature

Taking the previous example of secret.json, the base64url encoding of the secret content is illustrated below:

$ cat secret.json | basenc --base64url -w0

ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvbXlzZWNyZXQvc2VjcmV0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cgo=

The sealed secret will be as shown below:

sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvbXlzZWNyZXQvc2VjcmV0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cgo=.fakesignature

Now we create the K8s secret using the base64url encoded sealed secret as shown below:

$ kubectl create secret generic mysecret --from-literal='secret=sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvbXlzZWNyZXQvc2VjcmV0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cgo=.fakesignature'

Now, we can use the secret in a K8s workload (deployed as CoCo) just like any other K8s secret — either as an environment variable or as a volume mount.

The unsealed secret will only be available inside the workload.

Here is a sample pod deployed as CoCo (note the runtimeClass). The secret will be available in plaintext inside the container under /sealed/myvalue or as the environment variable PROTECTED_SECRET.

apiVersion: v1
kind: Pod
metadata:
  name: app
  labels:
    app: app
  annotations:   
spec:
  runtimeClassName: kata-cc-snp
  containers:
    - name: app
      volumeMounts:
        - name: secret-volume
          mountPath: "/sealed/myvalue"   
      - name: PROTECTED_SECRET
        valueFrom:
          secretKeyRef:
            name: sealed-secret
            key: secret
      image: quay.io/fedora/fedora:39
      command:
        - sh
        - -c
        - |
          env
          echo "PROTECTED_SECRET=$PROTECTED_SECRET"
          sleep "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
  volumes:
    - name: secret-volume
      secret:
        secretName: mysecret  

As you can see, sealed secrets make it very easy to deploy your existing workloads as CoCo with minimal changes.

One current limitation of the sealed secrets feature is that to mount the sealed secret as a volume; you’ll need to prefix the mount path with /sealed. I have raised a PR to remove this limitation.

I hope this feature helps you to deploy your workload as CoCo and increase the security of your sensitive data and code. If you run into any issues or have suggestions for improvement feel free to raise an issue here.