Different ways to debug your Kubernetes apps

This article will briefly discuss what I typically use for debugging Kubernetes apps. I hope you will find it helpful.

Using plain old “exec”

$ kubectl exec -it -c <container> <pod-name> -- /bin/sh

This approach will not work for the environments where “exec” is disabled or for container images without a shell.

Let’s look at other approaches.

Using a sidecar

Enable process namespace sharing so that you can view processes in the app container from your debug container. You’ll generally need this for debugging.

An example POD YAML using a sidecar image with network debugging tools.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  shareProcessNamespace: true
  containers:
  - name: nginx
    image: bitnami/nginx
  - name: debug
    image: wbitt/network-multitool
    securityContext:
      capabilities:     
        add: ["NET_ADMIN"]   

Using ephemeral containers

This is the latest Kubernetes feature and the most preferred.

You can create a debug container on the fly without pre-provisioning a sidecar. More details are available in the official docs — https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers.

Let’s see this in action. In the following example, I create a debug container with perf tools.

Here is my pod YAML:

$ cat > nginx.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  shareProcessNamespace: true  
  containers:
  - name: nginx
    image: bitnami/nginx
EOF
$ kubectl apply -f nginx.yaml
$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          2s

Now I create an ephemeral container by using kubectl debug

$ kubectl debug --image=quay.io/bpradipt/perf-amd64 -it --share-processes=true nginx -- /bin/bash
Defaulting debug container name to debugger-mxtpf.
If you don't see a command prompt, try pressing enter.
bash-5.1# 

Now you can perform your debugging tasks.

If you are wondering what’s happening behind the scenes, then the best place to look at is the POD spec — kubectl get pod nginx -o yaml

You’ll see a new section added for ephemeral containers, as shown below:

...  
  ephemeralContainers:
  - command:
    - /bin/bash
    image: quay.io/bpradipt/perf-amd64
    imagePullPolicy: Always
    name: debugger-mxtpf
    resources: {}
    stdin: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
...

You can also create an entirely new debug POD as shown below:

$ kubectl debug --image=quay.io/bpradipt/perf-amd64 -it --share-processes=true --copy-to=debug-nginx nginx -- /bin/bash
Defaulting debug container name to debugger-mxtpf.
If you don't see a command prompt, try pressing enter.
bash-5.1#

You’ll see a new debug POD created.

$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
debug-nginx   2/2     Running   0          8s
nginx         1/1     Running   0          8m32s

I’m sure you’ll find ephemeral containers handy for debugging.
If you have some cool tricks, please do share.