Karpenter Containerd Runtime Mismatch With Datadog Docker Daemon

Datadog unable to post the Payload from the nodes provisioned by Karpenter. It prompts with the below error complaining about Multiple Mount points in your Kubernetes Pod.

2023–01–03 18:37:16 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:125 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:135) | Unable to get disk metrics for /host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c5
ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links:
‘/host/var/run/containerd/io.containerd.runtime.v2.task/k8s.io/84b24aadc886673856bde8c 5ceb172658ec8e4f6d2d30e13b4c7ed2528da00af/rootfs/host/proc/sys/fs/binfmt_misc’. You can exclude this mountpoint in the settings if it is invalid.

Debug Steps:

  • Connect to Right Context path of your Kubernetes Cluster.
  • Fetch the Datadog Pod name installed in your Datadog namespace.
  • kubectl logs datadog-xxxx -n datadog -c agent
  • Login to your pod with kubectl exec -i -t -n datadog datadog-xxxx — /bin/sh command
  • Check the File System mounts
  • This clearly shows that the Nodes provisioned using Karpenter uses containerd as Runtime environment while creating containers.
  • kubectl get nodes -o wide — this command also shows the Container Runtime (dockerd / containerd)

Check the Datadog Agent Runtime environment. In this case, Datadog agent uses dockerd and Karpenter uses Containerd

Modify the Karpenter Provisioner configuration:
kubectl edit provisioner default

containerRuntime: dockerd

Ref: https://karpenter.sh/v0.18.1/provisioner/

After making the changes, you need to do a rolling restart again to spawn up a new container with dockerd container runtime.
kubectl rollout restart deploy <name>

Leave a Comment

Your email address will not be published. Required fields are marked *