Karpenter Containerd Runtime mismatch with Datadog docker daemon
Issue:
Datadog unable to post the Payload from the nodes provisioned by Karpenter. It prompts with the below error complaining about Multiple Mount points in your Kubernetes Pod.
2023–01–03 18:37:16 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:125 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:135) | Unable to get disk metrics for /host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c5
ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links:
‘/host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c 5ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc’. You can exclude this mountpoint in the settings if it is invalid.
Debug Steps:
- Connect to Right Context path of your Kubernetes Cluster.
- Fetch the Datadog Pod name installed in your Datadog namespace.
- kubectl logs datadog-xxxx -n datadog -c agent
- Login to your pod with kubectl exec -i -t -n datadog datadog-xxxx — /bin/sh command
- Check the File System mounts

- This clearly shows that the Nodes provisioned using Karpenter uses containerd as Runtime environment while creating containers.
- kubectl get nodes -o wide — this command also shows the Container Runtime (dockerd / containerd)
Solution:
Check the Datadog Agent Runtime environment. In this case, Datadog agent uses dockerd and Karpenter uses Containerd
Modify the Karpenter Provisioner configuration:
kubectl edit provisioner default
kubeletConfiguration:
containerRuntime: dockerd
Ref: https://karpenter.sh/v0.18.1/provisioner/
After making the changes, you need to do a rolling restart again to spawn up a new container with dockerd container runtime.
kubectl rollout restart deploy <name>