How to Size a memory-backed Kubernetes emptyDir Volume

A Kubernetes emptyDir volume is a directory that exists on the local node’s filesystem with no contents. These volumes are stored either on the node’s backing disk storage or memory. This type of volume is typically used as a local cache or a means to share data between different containers of a POD.

When emptyDir is memory-backed (emptyDir.medium field is set to “Memory”), the volume is backed by a tmpfs filesystem, which means they will be stored in memory and not on the backing storage of the node.

Normally when using memory-backed emptyDir volume the size is directly proportional to the amount of memory available on the node. It’s defaulted to 50% of the memory on the Linux node. You can read more about the tmpfs filesystem and its behaviour in the following doc.

This is an example POD YAML with memory-backed emptyDir volume

apiVersion: v1
kind: Pod
metadata:
  name: buildah-emptydir 
spec:
  containers:
    - name: buildah
      image: quay.io/buildah/stable:v1.23.1
      command: ["sleep", "infinity"]  
      volumeMounts:
      - mountPath: /var/lib/containers
        name: container-storage
  volumes:
  - name: container-storage
    emptyDir:
     medium: Memory

This is a sample output from my K8s 1.21 cluster, where you can see the volume size to be ~50% of the total node memory.

$ k get nodes
NAME            STATUS   ROLES                  AGE     VERSION
test-master-0   Ready    control-plane,master   9m46s   v1.21.10
$ k get pods
NAME               READY   STATUS    RESTARTS   AGE
buildah-emptydir   1/1     Running   0          5m43s
$ k exec -it buildah-emptydir -- bash
[root@buildah-emptydir /]# free -h
               total        used        free      shared  buff/cache   available
Mem:           5.8Gi       965Mi       889Mi       3.0Mi       4.0Gi       4.6Gi
Swap:             0B          0B          0B
[root@buildah-emptydir /]# df -h /var/lib/containers/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.0G     0  3.0G   0% /var/lib/containers

This poses a challenge if you want to ensure a specific size for your memory-backed emptyDir volumes irrespective of which node it runs. The strong coupling of default memory-backed volume size with the node that runs the pod is undesirable.

Let’s look at a few approaches that are available today to overcome this challenge.

Using the new Kubernetes SizeMemoryBackedVolumes feature gate

When this is enabled, you can specify a size for memory-backed volumes.

This feature gate is enabled by default starting K8s version 1.22

Create a sample POD using the below YAML

apiVersion: v1
kind: Pod
metadata:
  name: buildah-emptydir 
spec:
  containers:
    - name: buildah
      image: quay.io/buildah/stable:v1.23.1
      command: ["sleep", "infinity"]  
      volumeMounts:
      - mountPath: /var/lib/containers
        name: container-storage
  volumes:
  - name: container-storage
    emptyDir:
     medium: Memory
     sizeLimit: 1Gi

Verify the size of the ephemeral volume.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0G     0  1.0G   0% /var/lib/containers

Note that when this feature gate is enabled and you are not specifying the “sizeLimit” value then the entire node memory is available. The following is an example from my worker node having a 12G memory capacity.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            12G     0   12G   0% /var/lib/containers

If you are following best practices and assigning resource limits to the POD, then you won’t face this issue as shown for the example YAML below:

apiVersion: v1
kind: Pod
metadata:
  name: buildah-emptydir
spec:
  containers:
    - name: buildah
      image: quay.io/buildah/stable:v1.23.1
      command: ["sleep", "infinity"]
      resources:
        limits:
          memory: 2Gi
      volumeMounts:
      - mountPath: /var/lib/containers
        name: container-storage
  volumes:
  - name: container-storage
    emptyDir:
     medium: Memory

The tmpfs mount is restricted to “2G”- the assigned memory limit for the container.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.0G     0  2.0G   0% /var/lib/containers

Without using the SizeMemoryBackedVolumes feature gate

One way is to explicitly mount tmpfs with the required size inside the container. However, you’ll need to use privileged or CAP_SYS_ADMIN capability. Also, this approach is only suitable when using a single container in a POD.

apiVersion: v1
kind: Pod
metadata:
  name: buildah-emptydir
spec:
  containers:
    - name: buildah
      image: quay.io/buildah/stable:v1.23.1
      resources:
        limits:
          memory: 2Gi
      command: ["sh", "-c"]
      args:
      - mkdir -p /var/lib/containers && 
        mount -t tmpfs -o size=1G tmpfs /var/lib/containers &&
        sleep infinity
      securityContext:
        privileged: true

If you are using or exploring Kata containers, then currently sizeLimit is not working due to a bug.

Hope the information in this blog helps with the effective usage of memory-backed emptyDir volumes in your environment.