vSphere Supervisor Manual Spherelet Certificate Renewal

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service
• vSphere with Tanzu
• VMware vSphere 7.0 with Tanzu
• Tanzu Kubernetes Runtime

Issue/Introduction

In a vSphere Supervisor environment, the spherelet service is installed on ESXi hosts for integration with Kubernetes.

The spherelet certificate has a default expiry of 1 year.

 

When the spherelet certificate is expired, the environment will continue to function but provisioning of new nodes or pods will become stuck.

• vSphere PodVMs will fail if the ESXi hosts are unable to run Spherelet correctly.
• WCP Supervisor Cluster upgrades may fail when the Spherelet certs are expired

 

The certmgr script from the following KB article is designed to rotate Supervisor cluster certificates and spherelet certificates on all ESXi hosts in the Supervisor cluster:

• Replace vSphere with Tanzu Supervisor Certificates

Please see the following KB article for steps to confirm on the status of the spherelet service in the affected ESXi host(s):

• vSphere Supervisor ESXi Hosts with Kubernetes status showing "Node is not healthy and is not accepting pods. Details Kubelet stopped posting node status."

 

This manual spherelet certificate renewal KB article is written to renew spherelet certificates when the above certmgr script fails to renew them.

Environment

vSphere 7.0 with Tanzu

 

vSphere 8.0 with Tanzu

Cause

The spherelet certificate has a default expiry of 1 year.

 

When the spherelet certificate is expired, the environment will continue to function but provisioning of new nodes or pods will become stuck.

• vSphere PodVMs will fail if the ESXi hosts are unable to run Spherelet correctly.
• WCP Supervisor Cluster upgrades may fail when the Spherelet certs are expired

Resolution

Confirm that the checks and steps in the following KB article do not resolve the spherelet certificate expiry:

• vSphere Supervisor ESXi Hosts with Kubernetes status showing "Node is not healthy and is not accepting pods. Details Kubelet stopped posting node status."
  
  

If certmgr fails to properly renew spherelet certificates, utilize the script attached to this KB to renew the expired spherelet certificates.

1. While logged into the vSphere web client, navigate to the Cluster object under Hosts and Clusters Inventory, and note down its managed object ID (MOID), which is its unique ID:
   • In the below example screenshot, the cluster object's MOID is domain-c1006
   • 
     
     
     
2. Navigate to each ESXi host under Hosts and Clusters Inventory and note down the associated managed object ID (MOID) as well:

   • In the below example screenshot, one of the ESXi host's MOID is host-1010

   • 
3. Upload the following attached script files to vCenter Server Appliance (VCSA) and ensure that these scripts are in the same folder:

   • rotate_spherelet_certs.sh

   • rotate_spherelet_certs.py
     
     
4. Change the permissions of the uploaded bash script file to be executable:

   • chmod +x rotate_spherelet_certs.sh
     
     

5. Run the rotate_spherelet_certs.sh bash script using the cluster ID and ESXi host ID collected in previous steps:

   • ./rotate_spherelet_certs.sh --cluster <cluster-MOID> --host <host-MOID>
     
     

6. Confirm that the spherelet certificates were successfully renewed:
   • From the Supervisor Cluster context, the ESXi hosts should now show as Ready:

     • kubectl get nodes
       
       NAME                    STATUS    ROLES                   AGE       VERSION
       <supervisor-dns-name-1> Ready     control-plane,master    ###d      v#.##.#
       <supervisor-dns-name-2> Ready     control-plane,master    ###d      v#.##.#
       <supervisor-dns-name-3> Ready     control-plane,master    ###d      v#.##.#
       <esxi-hostname-01>      Ready     agent                   ###d      v#.##.#-sph
       <esxi-hostname-02>      Ready     agent                   ###d      v#.##.#-sph
       <esxi-hostname-03>      Ready     agent                   ###d      v#.##.#-sph
       
       

   • While connected to the ESXi host where the spherelet certificate was renewed:
     • The certificate should no longer show as expired:

       • openssl x509 -text -in /etc/vmware/spherelet/client.crt | grep Not

       • openssl x509 -text -in /etc/vmware/spherelet/spherelet.crt | grep Not
         
         

     • The spherelet service is running without errors:

       • /etc/init.d/spherelet status
         
         YYYY-MM-DD HH:MM:SS,sss init.d/spherelet spherelet is running
         YYYY-MM-DD HH:MM:SS,sss init.d/spherelet spherelet is running

       • cat /var/log/spherelet.log
         
         

7. If certificates are not successfully renewed or there are still issues with spherelet, please reach out to VMware by Broadcom Technical Support referencing this KB article.