TKGm Management Kind Cluster Creation Fails When cgroup v1 Is Enabled

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

TKGm Management cluster creation fails during the creation of the kind clusters when cgroup v1 is enabled on Photon and Ubuntu jumpboxes.

Kubelet reports the below errors for the kind cluster failing to start:

GET https://tkg-kind-cs8n5qg1ptf3h99b3d2g-control-plane:6443/healthz?timeout=10s in 0 millisecondsI1017 20:28:16.759358 200 round_trippers.go:553] 
GET https://tkg-kind-cs8n5qg1ptf3h99b3d2g-control-plane:6443/healthz?timeout=10s in 0 milliseconds[kubelet-check] It seems like the kubelet isn't running or healthy.[kubelet-check]

The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.Unfortunately, an error has occurred:timed out waiting for the condition
This error is likely caused by:- The kubelet is not running- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

Kubelet logs for the kind cluster report the error:

kubelet[243]: E1017 23:37:36.280003     243 kubelet.go:1566] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: error validating root container [kubelet kubepods] : cgroup [\"kubelet\" \"kubepods\"] has some missing paths: /sys/fs/cgroup/systemd/kubelet.slice/kubelet-kubepods.slice"
systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

Environment

TKGm v2.5.2

Cause

It is required for cgroup v2 to be enabled for kind clusters therefore cluster creation fails when cgroup v1 is enabled on the jumpbox.

Resolution

To resolve this issue disable cgroup v1 and enable cgroup v2 on the jumpbox using the steps below for Photon OS or Ubuntu OS.

Verify the current cgroup setting by running the below command. If cgroup v1 is enabled the command output will be tmpfs.
# stat -fc %T /sys/fs/cgroup/
tmpfs

Photon OS 

1. To disable cgroup v1 and enable cgroup v2 on a Photon OS jumpbox update the /boot/systemd.cfg file with "systemd.unified_cgroup_hierarchy" and "systemd.legacy_systemd_cgroup_controller"

# Grub Environmental Block
systemd.unified_cgroup_hierarchy=1

2. Reboot the jumpbox, then verify the cgroup v2 has been enabled by running the below command and receiving output cgroup2fs.
# stat -fc %T /sys/fs/cgroup/
cgroup2fs

Ubuntu OS

1. To enable cgroup v2 on Ubuntu OS, update the /etc/default/grub file and append the "GRUB_CMDLINE_LINUX=" line:" with "systemd.unified_cgroup_hierarchy=1":

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"

2. Initiate the update by running the below command:
# sudo update-grub

3. Reboot the jumpbox, then verify the cgroup v2 has been enabled by running the below command and receiving output cgroup2fs.
# stat -fc %T /sys/fs/cgroup/
cgroup2fs