NSX and AKO Pods Restart Frequently on Supervisor in vCenter 8.0 Update 3g or older

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Symptoms
  • NSX (NCP) and Avi Kubernetes Operator (AKO) pods on the vSphere Supervisor cluster restart approximately every two hours.
  • The environment is running vCenter 8.0 Update 3g or older
  • In the "certmonitor.log" file on the Supervisor control plane nodes, entries indicate a certificate rotation is being triggered frequently:

    • Node ######## (schedext.cert expires 2026-10-28T##:##:##Z):
      ##:##:##Z  Detected certificate expiration within rotation threshold, running rotation
      ##:##:##Z  Certificate rotation finished successfully
      ##:##:##Z  kube-apiserver 2223.log: "Version: v1.29.7+vmware.wcp.1"  (new process satrts)

    • Checking the `schedext.cert` expiration (e.g., via `openssl`) shows the expiration date does not advance despite the logs claiming a successful rotation.

Environment

• vSphere 8.x

Cause

This issue is caused by a known software defect in the certificate lifecycle management of vCenter 8.0 Update 3. 

The system incorrectly identifies the "schedext.cert" certificate as nearing expiration.  While the rotation process is triggered and completes, a known issue prevents the updated certificate from persisting.  Because the "old" expiration date remains in place, the Supervisor continues to detect a pending expiration and triggers a new rotation and pod restart cycle every two hours.

Resolution

• This issue is resolved in vCenter 8.0 U3h and later releases.  Broadcom recommends updating the vCenter to this version in order to apply the permanent fix.
• Workaround
  • If an update to the vCenter is not immediately possible, a rotation of the Supervisor certificates will be necessary to force the persistence of the new certificate expiration date:
    
    • Use the "certmgr" utility on the vCenter to rotate the Supervisor certificates.
      
      • The "certmgr" utility can be downloaded from KB article 322994, which also provides detailed instructions on using the utility:
        • Replace vSphere with Tanzu Supervisor Certificates

 

Additional Information

• Replace vSphere with Tanzu Supervisor Certificates