Expired SSL Certificate on Port 10258 for vsphere-cloud-controller-manager

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Security scanners report an expired SSL certificate on TKG nodes specifically on port 10258.
• Standard Kubelet certificate rotations (port 10250) and kapp-controller rotations have been completed, but port 10258 still presents an expired certificate.
• Checking the certificate via openssl reveals a "Not After" date in the past
  
  openssl s_client -connect <NODE_IP>:10258 < /dev/null 2>/dev/null | openssl x509 -text -noout | grep "Not After"

Environment

TKG 2.5.2

Cause

• Port 10258 is utilized by the vsphere-cloud-controller-manager (CCM) pod.
• This component acts as the interface between Kubernetes and vCenter for node IP management and storage provisioning.
• The CCM pod generates a self-signed, in-memory certificate during process initialization with a hardcoded 365-day validity period.
• If a cluster or its nodes have not been restarted or upgraded for more than a year this in-memory certificate expires.

Resolution

Rotate the certificate must be rotated by forcefully restarting the CCM pods.This triggers the initialization logic to issue a new certificate with 365 days validity.

Note: This is a safe operation that only briefly pauses vCenter infrastructure synchronization.

1. Trigger a rolling restart of the Cloud Controller Manager DaemonSet.
   

   kubectl rollout restart daemonset vsphere-cloud-controller-manager -n kube-system

2. Verify Pod Initialization: Wait approximately 60 seconds for the new pods to reach a Running state
   
   kubectl get pods -n kube-system -l component=cloud-controller-manager
   
   
3. Validate the Certificate: Run the network check again to confirm the new expiration date 
   
   openssl s_client -connect <NODE_IP>:10258 < /dev/null 2>/dev/null | openssl x509 -text -noout | grep "Not After"