From Sandboxed Containers to Confidential Containers — Part-2

In the first part of this series, we learnt about sandboxed (virtualised) containers and how it provides an added layer of security to container workloads by running the pod in a virtual machine (VM). We also examined how sandboxed containers, implemented in Kata containers, provide the foundation for confidential containers. After that, we briefly discussed the limitations of sandboxed and confidential containers as a bare metal server-only solution.

This blog will look at how we enabled sandboxed and confidential containers to run on any footprint without needing bare metal servers by introducing the “peer-pods” approach.

In the “peer-pods” approach, we extended the Kata Containers runtime to manage the virtual machine (VM) lifecycle using cloud provider APIs or third-party hypervisor APIs. Contrast this with the standard approach of using QEMU or cloud hypervisor to launch VMs on bare metal servers or using nested virtualisation to launch VMs on VMs.

The diagram below shows the two approaches side-by-side by taking the example of confidential containers— the standard approach of creating the confidential VMs on the worker node vs the “peer-pods” approach of creating the confidential VM external to the worker node by invoking cloud provider APIs (e.g. AWS, Azure etc.).

Confidential containers using the standard approach Vs the peer-pods approach

For ease of understanding, let me now introduce two terms — local hypervisor and remote hypervisor.

The local hypervisor is qemu or cloud-hypervisor, which creates a (confidential) VM on the worker node for every pod creation request. This is the approach we saw for sandboxed and confidential containers in the first part of the series.

The remote hypervisor uses a cloud provider or 3rd party hypervisor APIs to create a (confidential) VM external to the worker node for every pod creation request. We refer to the usage of the remote hypervisor to create the VM as the “peer-pods” approach, where the VM is a peer to the worker node. This is the new approach to creating confidential containers using the confidential VM services provided by cloud providers.

The following diagram shows the two hypervisor approaches side-by-side for easier understanding.

Kata with local hypervisor (standard approach) Vs remote hypervisor (peer-pods approach)

Now with this basic understanding of the two approaches, let’s briefly understand what happens when you create or delete a pod.

I’m using two different Kubernetes (K8s) runtimeClasses — kata and kata-remote, to handle the configuration differences of the Kata runtime for the local and remote hypervisors. You can read more details on using runtimeClass for different container runtimes and configurations in the following blog.

What happens when you create a pod?

The following diagrams show the flow of creating a sandboxed or confidential pod using the standard approach (local hypervisor) and the “peer-pods” (remote hypervisor) approach in AWS.

Sandboxed or Confidential pod creation using Kata local hypervisor (standard approach)

Sandboxed or Confidential pod creation using Kata remote hypervisor (peer-pods approach)

Let’s take a closer look at the differences between these solutions for creating a pod:

• The first steps for both approaches are the same, with Kubelet invoking CRIO/containerd, which then calls the Kata runtime.
• In the standard approach, the Kata runtime uses QEMU/KVM to create a VM on the worker node and runs the pod inside the VM.
• In the “peer-pods” approach, the Kata runtime uses EC2 APIs to create a VM external to the worker node and runs the pod inside the VM.
• The difference you immediately see between these solutions is the networking portion. For the “peer-pods” approach, we create a tunnel from the worker node to the pod VM, transparently extending the pod network from the worker node to the external pod VM.

What happens when you delete a pod?

The following diagrams show the flow of deleting a sandboxed or confidential pod using the standard approach (local hypervisor) and the “peer-pods” (remote hypervisor) approach in AWS.

Sandboxed or Confidential pod deletion using Kata local hypervisor (standard approach)

Sandboxed or Confidential pod deletion using Kata remote hypervisor (peer-pods approach)

Let’s take a closer look at the differences between these solutions for deleting a pod:

• The first steps for both approaches are the same, with Kubelet invoking CRIO/containerd and then calling the Kata runtime.
• In the standard approach, the Kata runtime uses QEMU/KVM to delete the VM on the worker node that is running the pod.
• In the “peer-pods” approach, the Kata runtime uses EC2 APIs to delete the VM that is running the pod.

For ease of understanding, I have avoided going into too much detail. Please look at the project’s GitHub page here for more information and also to get involved.

Conclusion

In the two-part series, we explored the journey from sandboxed (virtualised) containers to confidential containers built on the standard foundation of Kata containers runtime. We also looked at the “peer-pods” approach, which allows spinning up pod VMs as peers to the worker node (in response to pod creation requests) to leverage the confidential computing capabilities provided by the cloud providers.

The entire confidential container work is happening in the open under the CNCF confidential containers project. Please feel free to join us and help shape the project. For any questions or clarifications, please reach out to me.

The diagrams first appeared in the following blog describing “peer-pods” availability for OpenShift.