How to Size a memory-backed Kubernetes emptyDir Volume

Pradipta Banerjee
3 min readFeb 25, 2022

--

A Kubernetes emptyDir volume is a directory that exists on the local node’s filesystem with no contents. These volumes are stored either on the node’s backing disk storage or memory. This type of volume is typically used as a local cache or a means to share data between different containers of a POD.

When emptyDir is memory-backed (emptyDir.medium field is set to “Memory”), the volume is backed by a tmpfs filesystem, which means they will be stored in memory and not on the backing storage of the node.

Normally when using memory-backed emptyDir volume the size is directly proportional to the amount of memory available on the node. It’s defaulted to 50% of the memory on the Linux node. You can read more about the tmpfs filesystem and its behaviour in the following doc.

This is an example POD YAML with memory-backed emptyDir volume

apiVersion: v1
kind: Pod
metadata:
name: buildah-emptydir
spec:
containers:
- name: buildah
image: quay.io/buildah/stable:v1.23.1
command: ["sleep", "infinity"]
volumeMounts:
- mountPath: /var/lib/containers
name: container-storage
volumes:
- name: container-storage
emptyDir:
medium: Memory

This is a sample output from my K8s 1.21 cluster, where you can see the volume size to be ~50% of the total node memory.

$ k get nodes
NAME STATUS ROLES AGE VERSION
test-master-0 Ready control-plane,master 9m46s v1.21.10
$ k get pods
NAME READY STATUS RESTARTS AGE
buildah-emptydir 1/1 Running 0 5m43s
$ k exec -it buildah-emptydir -- bash[root@buildah-emptydir /]# free -h
total used free shared buff/cache available
Mem: 5.8Gi 965Mi 889Mi 3.0Mi 4.0Gi 4.6Gi
Swap: 0B 0B 0B
[root@buildah-emptydir /]# df -h /var/lib/containers/
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.0G 0 3.0G 0% /var/lib/containers

This poses a challenge if you want to ensure a specific size for your memory-backed emptyDir volumes irrespective of which node it runs. The strong coupling of default memory-backed volume size with the node that runs the pod is undesirable.

Let’s look at a few approaches that are available today to overcome this challenge.

Using the new Kubernetes SizeMemoryBackedVolumes feature gate

When this is enabled, you can specify a size for memory-backed volumes.

This feature gate is enabled by default starting K8s version 1.22

Create a sample POD using the below YAML

apiVersion: v1
kind: Pod
metadata:
name: buildah-emptydir
spec:
containers:
- name: buildah
image: quay.io/buildah/stable:v1.23.1
command: ["sleep", "infinity"]
volumeMounts:
- mountPath: /var/lib/containers
name: container-storage
volumes:
- name: container-storage
emptyDir:
medium: Memory
sizeLimit: 1Gi

Verify the size of the ephemeral volume.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.0G 0 1.0G 0% /var/lib/containers

Note that when this feature gate is enabled and you are not specifying the “sizeLimit” value then the entire node memory is available. The following is an example from my worker node having a 12G memory capacity.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 12G 0 12G 0% /var/lib/containers

If you are following best practices and assigning resource limits to the POD, then you won’t face this issue as shown for the example YAML below:

apiVersion: v1
kind: Pod
metadata:
name: buildah-emptydir
spec:
containers:
- name: buildah
image: quay.io/buildah/stable:v1.23.1
command: ["sleep", "infinity"]
resources:
limits:
memory: 2Gi
volumeMounts:
- mountPath: /var/lib/containers
name: container-storage
volumes:
- name: container-storage
emptyDir:
medium: Memory

The tmpfs mount is restricted to “2G”- the assigned memory limit for the container.

[root@buildah-emptydir /]# df -h /var/lib/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 2.0G 0 2.0G 0% /var/lib/containers

Without using the SizeMemoryBackedVolumes feature gate

One way is to explicitly mount tmpfs with the required size inside the container. However, you’ll need to use privileged or CAP_SYS_ADMIN capability. Also, this approach is only suitable when using a single container in a POD.

apiVersion: v1
kind: Pod
metadata:
name: buildah-emptydir
spec:
containers:
- name: buildah
image: quay.io/buildah/stable:v1.23.1
resources:
limits:
memory: 2Gi
command: ["sh", "-c"]
args:
- mkdir -p /var/lib/containers &&
mount -t tmpfs -o size=1G tmpfs /var/lib/containers &&
sleep infinity
securityContext:
privileged: true

If you are using or exploring Kata containers, then currently sizeLimit is not working due to a bug.

Hope the information in this blog helps with the effective usage of memory-backed emptyDir volumes in your environment.

--

--