Kubelet is not able to reconnect to the Kubernetes API server in vRA 8.x and vRO 8.x

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• In vRA / vRO 8.x, Kubelet might not be able to reconnect to the Kubernetes API server
• The command journalctl -u kubelet --no-pager shows the following error.

  Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

• Once this happens, the corresponding node displays a Not Ready state when running kubectl get nodes.
  • As a result, some maintenance operations cannot be performed on the system and service availability may become degraded.

Environment

VMware vRealize Automation 8.x
VMware vRealize Orchestrator 8.x

Cause

This is caused by a well-known bug in the Kubernetes API client library that is most often observed during high system load, on system startup, or after brief network interruption events.

Resolution

To resolve the issue upgrade to a major version of vRA / vRO that includes Kubernetes 1.20 or later.

To resolve this issue, upgrade to vRealize Automation 8.5

Workaround:

Temporary

1. Restart kubelet on the node in question with

   systemctl restart kubelet

Persistent

Install an automated monitor by following these steps:

For each vRA / vRO appliance in the cluster:

1. Take simultaneous VM snapshots without memory.
2. Run the following steps
   1. Download the attached file located under the Attachment(s) section titled 84236.tar.gz.
   2. Upload 84236.tar.gz to /root
   3. Run this command

      (tar -xvf 84236.tar.gz -C /tmp && chmod a+x /tmp/2722672-install-kubelet-monitor.sh && /tmp/2722672-install-kubelet-monitor.sh); rm -f /tmp/2722672-install-kubelet-monitor.sh

3. The patch will automatically validate if the monitor is working successfully.

Note: If the validation fails, contact VMware Support