vRops cAdvisor pods failing to collect filesystem stats with error unable to count inodes

Products

• VMware

Issue/Introduction

Symptoms:

Issue Description

• The cAdvisor pods are permanently in running state,0 restarts but with failure to update system stats due to permission issue on collecting system inodes as shown in Pods logs below

The logs are showing errors.

$ kubectl get pods -n pks-system
NAME READY STATUS RESTARTS AGE
...
vrops-cadvisor-8w552 1/1 Running 0 41d
vrops-cadvisor-bqbgz 1/1 Running 0 49d
vrops-cadvisor-wth28 1/1 Running 0 41d

$ kubectl logs vrops-cadvisor-wth28 -n pks-system | head
W0330 10:29:27.941769 1 manager.go:288] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
W0330 11:15:01.327593 1 container.go:586] Failed to update stats for container "/kubepods/burstable/podd75df460-2481-4603-b426-188388671582": /sys/fs/cgroup/cpuset/kubepods/burstable/podd75df460-2481-4603-b426-188388671582/cpuset.cpus found to be empty, continuing to push stats
E0330 11:17:22.167483 1 fsHandler.go:114] failed to collect filesystem stats - rootDiskErr: unable to count inodes for part of dir /rootfs/var/vcap/store/docker/docker/overlay2/8bb564180dbc1ccef2daba9896e27119b23d2fff713175c751323f07b5524e3a/diff: open /rootfs/var/vcap/store/docker/docker/overlay2/8bb564180dbc1ccef2daba9896e27119b23d2fff713175c751323f07b5524e3a/diff/var/cache/nginx/client_temp: permission denied, extraDiskErr: <nil>
E0330 11:19:22.167953 1 fsHandler.go:114] failed to collect filesystem stats - rootDiskErr: unable to count inodes for part of dir /rootfs/var/vcap/store/docker/docker/overlay2/8bb564180dbc1ccef2daba9896e27119b23d2fff713175c751323f07b5524e3a/diff: open /rootfs/var/vcap/store/docker/docker/overlay2/8bb564180dbc1ccef2daba9896e27119b23d2fff713175c751323f07b5524e3a/diff/var/cache/nginx/client_temp: permission denied, extraDiskErr: <nil>

Cause

Container Permissions running inside the cAdvisor pod does not have enough privilege  to count the inodes of the filesystem

Resolution

This issue will be fixed in TKGi versions 1.12.8 and 1.13.6 and 1.14.1

Workaround:

Current workaround for is to add securityContext: {privileged: true} to the container
spec.securityContext:
privileged: true

Additional Information

For Using VMware vRealize Operations Management Pack for Container Monitoring

You can monitor Tanzu Kubernetes Grid Integrated Edition Kubernetes clusters with VMware vRealize Operations Management Pack for Container Monitoring.

To integrate Tanzu Kubernetes Grid Integrated Edition with VMware vRealize Operations Management Pack for Container Monitoring, you must deploy a container running cAdvisor in your TKGI deployment.

cAdvisor is an open source tool that provides monitoring and statistics for Kubernetes clusters.

To deploy a cAdvisor container:

1. Select In-Cluster Monitoring.
2. Under Deploy cAdvisor, select Yes.
3. Click Save.

For more information about integrating this type of monitoring with TKGI, see the VMware vRealize Operations Management Pack for Container Monitoring User Guide https://docs.vmware.com/en/Management-Packs-for-vRealize-Operations-Manager/1.4/container-monitoring/GUID-BD6B5510-4A16-412D-B5AD-43F74C300C91.html