Karpenter Containerd Runtime mismatch with Datadog docker daemon

 In Uncategorized

Issue:
Datadog unable to post the Payload from the nodes provisioned by Karpenter. It prompts with the below error complaining about Multiple Mount points in your Kubernetes Pod.

2023–01–03 18:37:16 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:125 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:135) | Unable to get disk metrics for /host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c5
ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links:
‘/host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c 5ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc’. You can exclude this mountpoint in the settings if it is invalid.

Debug Steps:

  • Connect to Right Context path of your Kubernetes Cluster.
  • Fetch the Datadog Pod name installed in your Datadog namespace.
  • kubectl logs datadog-xxxx -n datadog -c agent
  • Login to your pod with kubectl exec -i -t -n datadog datadog-xxxx — /bin/sh command
  • Check the File System mounts
  • This clearly shows that the Nodes provisioned using Karpenter uses containerd as Runtime environment while creating containers.
  • kubectl get nodes -o wide — this command also shows the Container Runtime (dockerd / containerd)

Solution:
Check the Datadog Agent Runtime environment. In this case, Datadog agent uses dockerd and Karpenter uses Containerd

Modify the Karpenter Provisioner configuration:
kubectl edit provisioner default

kubeletConfiguration:
containerRuntime: dockerd

Ref: https://karpenter.sh/v0.18.1/provisioner/

After making the changes, you need to do a rolling restart again to spawn up a new container with dockerd container runtime.
kubectl rollout restart deploy <name>

Recent Posts
Learn Devops

Become a Devops Engineer in 3 months