Manual method to replace expired Self-Signed vip.crt certificate in Supervisor cluster
search cancel

Manual method to replace expired Self-Signed vip.crt certificate in Supervisor cluster

book

Article ID: 370691

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service vSphere with Tanzu

Issue/Introduction

  • The /etc/vmware/wcp/tls/vip.crt certificate marked as expired and not able to connect securely to the Supervisor Cluster API Endpoint. 

  • Navigate to the vSphere Client in a web browser:
    • For vSphere 8.x and lower, under Workload Management > Supervisor > "Supervisor Name" > Configure > Certificates, you see that the Workload Platform Management certificate has expired:

    • For vSphere 9.x and higher, under Supervisor Management > Supervisor > "Supervisor Name" > Configure > Certificates, you see that the Supervisor Management certificate has expired.
  • When attempting to log into the supervisor you receive the error:

Get "https://##.##.##.##:443/api/v1/namespaces/kube-public/configmaps/pinniped-info": tls: failed to verify certificate: x509: certificate is valid for ###.###.###.###, not ##.##.##.##

  • When querying for cert using “echo | openssl s_client -showcerts -connect <SUPERVISOR_IP>:443 2>/dev/null | openssl x509 -noout -text “, an old supervisor is listed in SAN.

Environment

  • VMware vCenter server 9.x
  • VMware vCenter server 8.x

Cause

The self-signed certificate located at /etc/vmware/wcp/tls/vip.crt has expired and the auto-rotation process has not been initiated.

This TLS certificate is served by the nginx proxy and runs in front of each CP VM on the workload network. It has a 1-year lifetime. Ideally, the vip.crt self-signed certificate will automatically rotate when wcpsvc detects it to be more than 50% of its lifetime (i.e., 6 months).

Resolution

1.  Validate vip.crt from the Supervisor node Control plane VM 

    1. Check if the TLS endpoint certificate is valid: 

      # find / -type f ( -name ".cert" -o -name ".crt" ) -print 2>/dev/null | egrep -v 'ca.crt$|ca-bundle.crt$|kubelet/pods|var/lib/containerd|run/containerd' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

      # openssl x509 -noout -text -in /etc/vmware/wcp/tls/vip.crt|grep After
      Not After : MM DD hh:mm:ss yyyy GMT
    2. Check if certificates are self-signed or custom certificates. Below is an example of a self-signed certificate
      # openssl x509 -in /etc/vmware/wcp/tls/vip.crt -text | grep CN

      Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vc-name.example.org, OU=VMware Engineering
      Subject: OU=vc-name.example.org, CN=kube-apiserver-domain-c7

2. Generate a new vip.crt from vCenter Server

    1. Navigate to the vSphere Client in a web browser:
      • For vSphere 8.x and lower, under Workload Management > Supervisor > "Supervisor Name" > Configure > Certificates, select the Actions dropdown for the Workload Platform Management certificate and click on Generate CSR

      • For vSphere 9.x and higher, under Supervisor Management > Supervisor > "Supervisor Name" > Configure > Certificates, select the Actions dropdown for the Supervisor Management certificate  and click on Generate CSR

    2. Once the CSR is generated, click Copy.

    3. SSH to the vCenter server and create a new file 'cert.csr' and copy the CSR content from GUI

      # vi cert.csr
    4. Generate a new cert using the below command.
      # /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile cert.csr --cert vip-renew.crt



3. Replace VIP certificate from vCenter Server GUI

    1. Copy the generated VIP certificate
      # cat vip-renew.crt
    2. Go back to vCenter Server GUI, navigate to the the Workload Management tab from Menu > Click on Supervisor Tab > Select the Supervisor cluster > Go to Configure Tab > Select Certificates > Select the Workload Platform Management certificate > Select Actions dropdown > Click on Replace Certificate.

    3. Upload the newly generated certificate file and click Replace Certificate to replace the certificate.
  •  

Additional Information

If vip.crt using the custom signed certificate, The Certificate needs to be signed by CA to renew following below document

Replace the VIP Certificate to Securely Connect to the Supervisor API Endpoint

Japanese version: vSphere with Tanzu で自己署名の vip.crt 証明書を手動で置き換える方法

 

When the goal is to ensure that Kubernetes API endpoint certificate is signed by the chosen Certificate Authority.

Option 1: Using Workload Platform Management > Actions > Generate CSR with a CA 

  • When one generates a CSR using the vCenter UI, the system generates the private key internally and handles the pairing for you. Because that key is managed within vCenter, it is not exported for download.
  • They then have to submit the vCenter-generated CSR to your Certificate Authority (CA) and retrieve the signed certificate chain (Leaf + Intermediate CAs + Root CA).
  • When they "Replace Certificate" they Upload or paste the full PEM certificate chain into the dialog leaving "Provide custom private key" UNCHECKED.

Option 2: Customer-Managed Key & CSR

 To import a certificate signed by a customer's own CA authority, please note that they must:

  • Generate a new private key and CSR independently (outside of the vCenter UI).
  • Send the externally generated CSR to your chosen Certificate Authority for signing.
  • When they "Replace Certificate", upload the resulting signed certificate chain (Leaf + Intermediate CAs + Root CA) and the corresponding private key generated

By using this method, the customer retains the private key, and the system uses the key you provide to validate the certificate.