vSphere with Tanzu ESXi worker nodes showing 'Not Ready'

Products

• VMware vSphere ESXi
• vSphere with Tanzu

Issue/Introduction

• ESXi worker nodes show 'Not Ready' status within the supervisor (See: Connect to Supervisor Cluster Using the Tanzu CLI and vCenter SSO Authentication)

NAME                          STATUS     ROLES                        AGE   VERSION
Supervisor_UUID               Ready      control-plane,master         24d   v1.29.7+vmware.wcp.1
Supervisor_UUID               Ready      control-plane,master         24d   v1.29.7+vmware.wcp.1
Supervisor_UUID               Ready      control-plane,master         24d   v1.29.7+vmware.wcp.1
ESXi_FQDN                     NotReady   agent                        24d   v1.29.3-sph-c8e42be
ESXi_FQDN                     NotReady   agent                        24d   v1.29.3-sph-c8e42be
ESXi_FQDN                     NotReady   agent                        24d   v1.29.3-sph-c8e42be
ESXi_FQDN                     NotReady   agent                        24d   v1.29.3-sph-c8e42be

• The ESXi host Spherelet service is not running

[user@ESXi_FQDN :~] /etc/init.d/spherelet status
YYYY-MM-DD HH:MM:SS.### init.d/spherelet spherelet init script invoked via the following hierarchy
YYYY-MM-DD HH:MM:SS.### init.d/spherelet 2880521: -sh
YYYY-MM-DD HH:MM:SS.### init.d/spherelet 2880518: /bin/sshd-session -i -R
YYYY-MM-DD HH:MM:SS.### init.d/spherelet 2880494: /bin/sshd-session -i -R
YYYY-MM-DD HH:MM:SS.### init.d/spherelet 2098773: /usr/lib/vmware/busybox/bin/busybox inetd /var/run/inetd.conf
YYYY-MM-DD HH:MM:SS.### init.d/spherelet 2097869: /bin/init
YYYY-MM-DD HH:MM:SS.### init.d/spherelet Log fetcher support: True
YYYY-MM-DD HH:MM:SS.### init.d/spherelet Log fetcher size: 460
YYYY-MM-DD HH:MM:SS.### init.d/spherelet spherelet is not running
YYYY-MM-DD HH:MM:SS.### init.d/spherelet spherelet is not running

• Attempting to start Spherelet service fails 
• The ESXi Spherelet.log contains similar messages to the following:

YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108903]: time="YYYY-MM-DDTHH:MM:SS" level=info msg="spherelet started!\n"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108903]: time="YYYY-MM-DDTHH:MM:SS" level=error msg="BuildK8sClientConfig failed" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108903]: time="YYYY-MM-DDTHH:MM:SS" level=fatal msg="Error initializing Spherelet" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108938]: time="YYYY-MM-DDTHH:MM:SS" level=info msg="spherelet started!\n"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108938]: time="YYYY-MM-DDTHH:MM:SS" level=error msg="BuildK8sClientConfig failed" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108938]: time="YYYY-MM-DDTHH:MM:SS" level=fatal msg="Error initializing Spherelet" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108973]: time="YYYY-MM-DDTHH:MM:SS" level=info msg="spherelet started!\n"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108973]: time="YYYY-MM-DDTHH:MM:SS" level=error msg="BuildK8sClientConfig failed" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
YYYY-MM-DDTHH:MM:SS No(5) spherelet[2108973]: time="YYYY-MM-DDTHH:MM:SS" level=fatal msg="Error initializing Spherelet" error="invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"

• Files within the ESXi directory /etc/vmware/spherelet/ show 0 bytes

[root@ESXi_FQDN:/etc/vmware/spherelet] ls -ltr
-rw-r--r-T    1 root     root             0 MMM DD HH:MM spherelet.crt
-rw-r--r-T    1 root     root             0 MMM DD HH:MM spherelet.conf
-rw-r--r-T    1 root     root             0 MMM DD HH:MM server.key
-rw-r--r-T    1 root     root             0 MMM DD HH:MM kubelet-server-current.pem
-rw-r--r-T    1 root     root             0 MMM DD HH:MM kubelet-client-current.pem
-rw-r--r-T    1 root     root             0 MMM DD HH:MM client.key
-rw-r--r-T    1 root     root             0 MMM DD HH:MM client.crt

Environment

vSphere with Tanzu

Resolution

To get the ESXi worker nodes back in a 'Ready' ;

1. Place the impacted ESXi host in maintenance mode
2. Connect to the supervisor cluster
3. Run the following command within the supervisor to remove the host from the supervisor cluster
    kubectl delete node <esxi node name>
4. Exit the ESXi host from maintenance mode
5. Run 'kubectl get nodes' within the supervisor to confirm the ESXi worker node has successfully rejoined the cluster