TKGI kubelet logs show: "Failed to find cgroups of kubelet" err="cpu and memory cgroup hierarchy not unified."

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When looking at the kubelet.stderr.log log in a TKGI worker VM, the following log message is seen:

container_manager_linux.go:514] "Failed to find cgroups of kubelet" err="cpu and memory cgroup hierarchy not unified.  cpu: /, memory: /system.slice/runit.service"

 

This repeats every 5 minutes.

Environment

TKGI, all versions.

Cause

This is due to the TKGI vms using cgroup v1. In v1, the CPU and memory controllers live in separate hierarchies, instead of v2, where they are unified. You can validated this by looking here:

worker/0:/var/vcap/jobs/kubelet/bin# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-133-generic root=UUID=f0d3ca9d-dc04-45dc-914f-26b5d393dce1 ro vconsole.keymap=us net.ifnames=0 biosdevname=0 crashkernel=auto selinux=0 plymouth.enable=0 console=ttyS0,115200n8 earlyprintk=ttyS0 rootdelay=300 ipv6.disable=1 audit=1 cgroup_enable=memory swapaccount=1 apparmor=1 security=apparmor systemd.unified_cgroup_hierarchy=false

 

The last entry, "systemd.unified_cgroup_hierarchy=false", verifies that the hierarchy is not unified.

Resolution

There is no impact from these messages, so they can safely be ignored. In the future, noble stemcells will enable cgroup v2, so these messages should disappear at that time.

There is currently no ETA for TKGI compatibility with Noble Stemcells.