Best practices for running Build Pipelines in OpenShift sandboxed containers

In this blog, I’ll list some of the best practices when running build pipelines in Red Hat OpenShift sandboxed containers (OSC) based on Kata containers.

When running build pipelines in OSC, there are two critical considerations:

1. The Kata VM Size
2. The scratch space for builds
   - For example, /var/lib/containers inside the pod when invoking a container based build with Tekton, Gitlab etc. using buildah
   - Local directory inside the pod

Let’s look into some of the best practices around the two considerations listed above when running build pipelines in OpenShift clusters using OSC with KVM/Qemu hypervisor.

Ensure required Kata annotations to enable fuse device is enabled in CRIO config

On Kata worker node, check /etc/crio/crio.conf.d/50-kata to ensure the following line is present (this is default with OSC version starting 1.7.0):

allowed_annotations = [
      "io.kubernetes.cri-o.Devices",
]

The complete configuration file looks like this:

[crio.runtime.runtimes.kata]
  runtime_path = "/usr/bin/containerd-shim-kata-v2"
  runtime_type = "vm"
  runtime_root = "/run/vc"
  privileged_without_host_devices = true
  allowed_annotations = [
        "io.kubernetes.cri-o.Devices",
  ]

The io.kubernetes.cri-o.Devices annotation enables adding of /dev/fuse device in the Kata pod using a pod annotation. Otherwise the /dev/fuse device needs to be created in the Kata pod.

Setting CPU and Memory for the Kata pod VM

The Kata pod VM uses 1 vCPU and 2GB of RAM as default which might not be sufficient for your build pipelines. If you have existing pipeline jobs with resource requests and limits that you want to run using OSC then use the same request and limits. Kata hotplugs the required CPU and memory to the pod VM based on the limits set, but does not increase the number of networking and I/O threads on the host dedicated to the VM. In addition, a an upstream bug results in the pod VM not onlining all of the requested CPUs.

Another (preferred) alternative is to use pod annotations to change the default Kata VM CPU and memory.

Annotations increase the number of networking threads available to the VM to the number of CPUs, and can also be used to increase the number of I/O threads. CPUs added by means of annotation are made available at boot time and are not subject to the above bug. It is essential, however, that you not combine limits and annotations. Using a CPU limit in addition to the default_vcpus annotation will result in the VM having additional CPUs which it cannot fully use, while using a memory limit in addition to the default_memory annotation will likely result in pod failures due to out of memory conditions.

The following is an example pod manifest that can be used. It sets the default vCPUs to 4, memory to 4GB, and the I/O threads to 2 for the pod VM:

apiVersion: v1
kind: Pod
metadata:
  name: buildah-kata
  namespace: sandboxed-builds
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "4096"
    io.katacontainers.config.hypervisor.default_vcpus: "4"
    io.katacontainers.config.hypervisor.virtio_fs_extra_args: "[\"--thread-pool-size=2\" ]"
spec:
  containers:
  ...

Ensure you set the Kata pod VM size as required for your build pipelines.

Scratch space for builds

The container root filesystem in a Kata pod uses virtiofs. Likewise any persistent volume mount using “FileSystem” volumemode is mounted as a virtiofs share in a Kata pod. virtiofs is not an optimal solution for IO intensive builds which include running containerised builds inside the pod with buildah. We have not found to date any combination of virtiofs threadpoolsize, caching, or writeback which materially improves performance.

If you are using buildah (or similar) via Tekton, Gitlab etc to run containerised builds in the pod, then it’s preferred to use either block device or memory backed filesystem (tpmfs) for container root storage (eg /var/lib/containers for buildah).

The following sections explain how to use a tmpfs or a block device as container storage inside the Kata pod for use with buildah builds.

Using tmpfs (RAM) as container root storage inside the pod

It’s important to note that the default size of the tmpfs storage is 50% of the VM memory. In the example below the size of the /var/lib/containers mount point will be approx 3 GB.

Following is an example pod manifest. Note the annotation for /dev/fuse device. This ensures that the /dev/fuse device required by the default storage driver (overlayfs) for buildah is present in the container.

apiVersion: v1
kind: Pod
metadata:
  name: buildah-kata
  namespace: sandboxed-builds
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "6144"
    io.katacontainers.config.hypervisor.default_vcpus: "4"
    io.kubernetes.cri-o.Devices: "/dev/fuse"
spec:
  runtimeClassName: kata
  containers:
    - name: buildah
      image: quay.io/buildah/stable:latest     
      command: ["sh", "-c"]
      args:
      - mkdir -p /var/lib/containers &&
        mount -t tmpfs tmpfs /var/lib/containers &&
        && sleep infinity
      securityContext:
        privileged: true

Alternatively, you can use something like the following pod manifest when planning to use memory backed file system for the builds:

apiVersion: v1
kind: Pod
metadata:
  name: buildah-kata
  namespace: sandboxed-builds
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "6144"
    io.katacontainers.config.hypervisor.default_vcpus: "4"
    io.kubernetes.cri-o.Devices: "/dev/fuse"
spec:
  runtimeClassName: kata
  containers:
    - name: buildah
      image: quay.io/buildah/stable:latest      
      command: ["sh", "-c"]
      args:
      - sleep infinity
      volumeMounts:
        - name: container-storage
          mountPath: /var/lib/containers
      securityContext:
        privileged: true
  volumes:
    - name: container-storage
      emptyDir:
        medium: Memory

It is possible to increase the size of a memory backed emptyDir volume beyond the default 50% of the VM memory by using the following command.

mount -o remount,size=3G /var/lib/containers

However it’s important to then size the Kata pod VM memory accordingly.

Using block devices as container storage

In order to use raw block devices as persistent volumes (PV), one option is to use the Local Storage Operator to manage physical volumes on the worker node itself. Alternatively you can use Fibre Channel (FC) or iSCSI volumes or CSI drivers providing raw block volumes based on your setup.

Details on using local storage operator installation and steps to create persistent volumes using local disks on the worker node is described here — Persistent storage using local volumes

Assuming you have additional storage disks attached to your worker node you can create partitions of specific sizes depending on the container storage requirements for your build pipelines. For example if the container storage requirements for your build pipeline is 2GB, then create partitions of size 2 GB on the worker node and use the partition details to create local persistent volumes using the local storage operator.

When using block volume, you’ll need to create a filesystem on it before being able to use it.

In the example below the filesystem creation is handled in the initContainer. In practice you should build a helper container image with the required tools pre-installed and use the same as the initContainer image instead of installing the packages when running the initContainer.

apiVersion: v1
kind: Pod
metadata:
  name: buildah-kata
  namespace: sandboxed-builds
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "6144"
    io.katacontainers.config.hypervisor.default_vcpus: "4"
    io.kubernetes.cri-o.Devices: "/dev/fuse"
spec:
  runtimeClassName: kata
  initContainers:
    - name: buildah-init
      image: quay.io/fedora/fedora:latest
      command: ["sh", "-c"]
      args:
      - dnf install -y xfsprogs &&
        mkfs.xfs -f /dev/xvdb &&
        mkdir -p /var/lib/containers &&
        mount /dev/xvdb /var/lib/containers       
      securityContext:
        privileged: true
      volumeDevices:
        - name: data
          devicePath: /dev/xvdb
  containers:
    - name: buildah     
      image: quay.io/buildah/stable:latest      
      command: ["sh", "-c"]
      args:
      - mount /dev/xvdb /var/lib/containers &&
        sleep infinity
      securityContext:
        privileged: true
      volumeDevices:
        - name: data
          devicePath: /dev/xvdb
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: block-pvc

Note: You might have to label the device on the worker node.

chcon -vt container_file_t /host/path/to/device

Sometimes it’s not possible to change the app container entry point to mount the filesystem. In such a scenario you can use the approach shown in the following manifest. Note the use of sidecar container and shareProcessNamespace

apiVersion: v1
kind: Pod
metadata:
  name: buildah-kata
  namespace: sandboxed-builds
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "6144"
    io.katacontainers.config.hypervisor.default_vcpus: "4"
    io.kubernetes.cri-o.Devices: "/dev/fuse"
spec:
  runtimeClassName: kata
  shareProcessNamespace: true
  initContainers:
    - name: buildah-init
      image: quay.io/fedora/fedora:latest
      command: ["sh", "-c"]
      args:
      - dnf install -y xfsprogs &&
        mkfs.xfs -f /dev/xvdb &&
        mkdir -p /var/lib/containers &&
        mount /dev/xvdb /var/lib/containers       
      securityContext:
        privileged: true
      volumeDevices:
        - name: data
          devicePath: /dev/xvdb
      restartPolicy: Always
  containers:
    - name: buildah     
      image: quay.io/buildah/stable:latest      
      command: ["sh", "-c"]
      args:
      - sleep infinity
      securityContext:
        privileged: true
      env:
        - name: GET_DATA_PROC_DIR
          value: 'find /proc -maxdepth 1 -type d -regex "/proc/[0-9]*" | head -2 | tail -1'
      lifecycle:
        postStart:
          exec:
            command:
              - /bin/sh
              - -c
              - ln -s $(eval $GET_DATA_PROC_DIR)/root/var/lib/containers /var/lib/containers

  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: block-pvc

This cool trick is courtesy https://github.com/kubernetes/kubernetes/issues/6120#issuecomment-1623674602

Summary

We explored various methods for building individual containers using OpenShift sandboxed containers (OSC). You can extend these approaches to a typical CI/CD pipeline. Let me know if you are using something that is not listed here.