Karpenter Containerd Runtime mismatch with Datadog docker daemon

Posted January 9, 2023

Issue:
Datadog unable to post the Payload from the nodes provisioned by Karpenter. It prompts with the below error complaining about Multiple Mount points in your Kubernetes Pod.

2023–01–03 18:37:16 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:125 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:135) | Unable to get disk metrics for /host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c5
ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links:
‘/host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c 5ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc’. You can exclude this mountpoint in the settings if it is invalid.

Debug Steps:

Connect to Right Context path of your Kubernetes Cluster.
Fetch the Datadog Pod name installed in your Datadog namespace.
kubectl logs datadog-xxxx -n datadog -c agent
Login to your pod with kubectl exec -i -t -n datadog datadog-xxxx — /bin/sh command
Check the File System mounts

This clearly shows that the Nodes provisioned using Karpenter uses containerd as Runtime environment while creating containers.
kubectl get nodes -o wide — this command also shows the Container Runtime (dockerd / containerd)

Solution:
Check the Datadog Agent Runtime environment. In this case, Datadog agent uses dockerd and Karpenter uses Containerd

Modify the Karpenter Provisioner configuration:
kubectl edit provisioner default

kubeletConfiguration:
containerRuntime: dockerd

Ref: https://karpenter.sh/v0.18.1/provisioner/

After making the changes, you need to do a rolling restart again to spawn up a new container with dockerd container runtime.
kubectl rollout restart deploy <name>

sowndarya C