Master vSphere Supervisor Certificate Guide
search cancel

Master vSphere Supervisor Certificate Guide

book

Article ID: 323421

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

This article provides guidance on vSphere Supervisor and Workload/Guest Cluster certificate management. 

 
--------------------------------------------------------------------- 


Certificates for vSphere Supervisor are utilized in 4 places depending on where the authentication is being performed, these are: vCenter Server, Supervisor Cluster Nodes, Workload/Guest Cluster Nodes, ESXi hosts (used to support vSphere POD functionality)
 
There are 2 trust domains in vSphere Supervisor:
 
  • VC trust domain: The default signer for certificates in this trust domain is the VMware Certificate Authority (VMCA) built into vCenter Server
  • K8s trust domain: The default signer for certificates in this trust domain is the Kubernetes CA.
 
 
--------------------------------------------------------------------- 



WCP requires certificates for the following operations:
  • Server certificate for wcpsvc (running on vCenter Server) to talk to the Supervisor Cluster api-servers over the management network
  • Server certificate for the authproxy server endpoint hosted on Supervisor Cluster VM
  • Server certificate for the schedext endpoint hosted on Supervisor Cluster VM
  • Client certificate for the WCP Solution user used by authproxy to query WCP vAPI
 
Regular k8s operations:
  • Client certificates for kubelet to authenticate to the API server
  • Kubelet server certificates for the API server to talk to the kubelets
  • Server certificate for the API server endpoint (TLS over the workload network)
  • Client certificates for administrators of the cluster to authenticate to the API server
  • Client certificates for the API server to talk to the kubelets
  • Client certificate for the API server to talk to etcd
  • Client certificate/kubeconfig for the controller manager to talk to the API server
  • Client certificate/kubeconfig for the scheduler to talk to the API server.
  • Client and server certificates for the front-proxy
 
--------------------------------------------------------------------- 
 
The following commands can be used from SSH to Supervisor or Guest Clusters to identify certificate expiry timeframe:
 
find / -type f \( -name "*.cert" -o -name "*.crt" \)  -print 2>/dev/null | egrep -v 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

cat /etc/kubernetes/admin.conf | grep certificate-authority-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After

cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -text | grep After

Environment

vSphere Supervisor

Cause

VMware by Broadcom Kubernetes products adhere to default Kubernetes certificate expiration timeframes.

Resolution

Delineation of WCP certificates, usage and locations
 
 
On Supervisor Control Plane nodes:
 
 
WCP specific certificates      
Cert Path Signed By Used for Cert Lifetime
/etc/vmware/wcp/tls/vip.crt VMCA/Custom TLS certificate served by the nginx proxy
running in front of each CP VM on the workload network		 1 Year / Custom
/etc/vmware/wcp/tls/mgmt.crt K8s CA TLS certificate served by the nginx proxy
running in front of each CP VM on the management network		 1 Year
 
/etc/vmware/wcp/tls/ncp/lb-default.cert		  VMCA/Custom Certificate applied to Service IP's built on the Ingress network in NSX-T  
1 Year / Custom
 
/etc/vmware/wcp/tls/wcpusr.cert		 VMCA Client certificate for VC solution user for WCP 2 Years*
/etc/vmware/wcp/tls/schedext.cert self-signed TLS certificate served by schedext 2 Years
 
/etc/vmware/wcp/tls/authproxy.crt		 K8s CA TLS certificate served by authproxy 2 Years
/etc/vmware/wcp/tls/docker-reg.crt K8s CA TLS certificate served by the internal docker registry 2 Years
/etc/vmware/wcp/tls/wcpagent.cert VMCA TLS certificate for docker registry and authproxy. No long in use after 7.0 U1  
/etc/vmware/wcp/tls/authproxy-client.crt 
 K8s CA It provides authentication between nginx and authproxy. Renewed automatically by the cert-manager pod.
 2 Years
/etc/vmware/wcp/tls/pinniped.crt 
 K8s CA This is the CA certificate used by Pinniped and it's a source of trust when nginx proxies requests for /wcp/pinniped to Pinniped server. Renewed automatically by the cert-manager pod.
 2 years
 
 
Kubernetes internal certificates      
Cert Path Signed By Used for Cert Lifetime
/var/lib/kubelet/pki/kubelet.crt  K8s CA Currently not used. Kubelet serves "content" to metrics servers  
1 Year
/etc/kubernetes/pki/scheduler.crt  K8s CA Used to authenticate with the scheduler pod  1 Year
/etc/kubernetes/pki/apiserver.crt  K8s CA Used to authenticate with K8s API server  
1 Year
 
/etc/kubernetes/pki/apiserver-etcd-client.crt		  K8s CA Used by API server to authenticate with ETCD  
1 Year
 
/etc/kubernetes/pki/apiserver-kubelet-client.crt		  K8s CA Used by API server to authenticate with kubelet  
1 Year
/etc/kubernetes/admin.conf K8s CA Used to authenticate with K8s API server 1 Year
 
/etc/kubernetes/pki/front-proxy-client.crt		  K8s CA    
1 Year
 
/etc/kubernetes/pki/etcd/server.crt		  K8s CA Cert used for ETCD Server authentication   
1 Year
 
/etc/kubernetes/pki/etcd/peer.crt		  K8s CA Cert used for ETCD Peer server authentication  
1 Year
 
/etc/kubernetes/pki/etcd/healthcheck-client.crt		  K8s CA    
1 Year
 
/etc/kubernetes/pki/bootstrapper.crt		  K8s CA Used for initial cluster bootstrap and customization  
n/a
/etc/kubernetes/pki/front-proxy-ca.crt  K8s CA K8s Front Proxy certificate authority  
10 Year
/etc/kubernetes/pki/etcd/ca.crt  K8s CA K8s ETCD certificate authority  
10 Year
/etc/kubernetes/pki/ca.crt  K8s CA  K8s Cluster certificate authority  
10 Year
 
 
----------------------------------------------------------------------------------
 
 
On ESXi Hosts (Depending on Version/Deployment Type):
 
 
Cert Path Cert Lifetime
/etc/vmware/spherelet/spherelet.crt 1 Year
/etc/vmware/spherelet/client.crt 1 Year
 
 
----------------------------------------------------------------------------------
 
VKS Workload/Guest Cluster Certificates:
 
 
VKS Workload/Guest Cluster Control Plane VMs  
Cert Path Cert Lifetime
/var/lib/kubelet/pki/kubelet.crt 1 Year
/etc/kubernetes/pki/apiserver.crt 1 Year
/etc/kubernetes/pki/apiserver-etcd-client.crt 1 Year
/etc/kubernetes/pki/etcd/server.crt 1 Year
/etc/kubernetes/pki/etcd/peer.crt 1 Year
/etc/kubernetes/pki/etcd/healthcheck-client.crt 1 Year
/etc/kubernetes/pki/front-proxy-client.crt 1 Year
/etc/ssl/certs/extensions-tls.crt 10 Year
 


Workaround:
You can use the following kb to rotate the Certificates for Supervisor Control Plane VM's and ESXi hosts here: Replace vSphere with Tanzu Supervisor Certificates

VKS Workload/Guest Cluster Certificates can be rotated by upgrading the cluster. If they have expired, then you can follow this kb to rotate them Replace vSphere with Tanzu Guest Cluster Certificates




Additional Information

Impact/Risks:
If certificates expire on the Supervisor or Workload/Guest Clusters, access and management of the clusters will fail. Applications and pods that were originally running in the Workload clusters will continue to function.

NOTE: If your certificates expire, you can replace them with the certmgr script included in the below KB:
Powered by Wolken Software