Symptoms:
vSphere with Tanzu supervisor certificates or spherelet certificates have expired or are about to expire.
You can use the following command while connected via SSH into either of the Supervisor Control Plane VMs.
# find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)
Main vSphere with Tanzu Cert Guide
$ scp ./wcp_cert_manager.zip [email protected]:/root
Example Output:
The authenticity of host '192.168.111.135 (192.168.111.135)' can't be established.
ECDSA key fingerprint is SHA256:RkfHc8xvRJ8ihqMD1CTQeMXEPrYJ6yaNEOhwKpCbt3w.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.111.135' (ECDSA) to the list of known hosts.
VMware vCenter Server 7.0.3.01000
Type: vCenter Server with an embedded Platform Services Controller
Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00
# unzip wcp_cert_manager.zip
Archive: wcp_cert_manager.zip
inflating: certmgr
# ls -l
total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip
# ./certmgr certificates rotate
+------------------+------------------------------------------------------------------------------------------------------+-------+
| CONTROL PLANE IP | RESULT | ERROR |
+------------------+------------------------------------------------------------------------------------------------------+-------+
| 192.168.111.202 | +---------------------------------------------------------------------------------+----------------+ | |
| | | TASKS | OVERALL STATUS | | |
| | +---------------------------------------------------------------------------------+----------------+ | |
| | | +--------------------------------+------------------------------------+-------+ | ok | | |
| | | | TASK | RESULT | ERROR | | | | |
| | | +--------------------------------+------------------------------------+-------+ | | | |
| | | | backup certificates | /root/backups-16739895901776834456 | | | | | |
| | | | rotate etcd server certificate | | | | | | |
| | | | rotate api server etcd client | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate etcd peer certificate | | | | | | |
| | | | rotate etcd health check | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate api server certificate | | | | | | |
| | | | rotate kubelet client api | | | | | | |
| | | | server certificate | | | | | | |
| | | | rotate front proxy certificate | | | | | | |
| | | | rotate controller-manager | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate scheduler certificate | | | | | | |
| | | | rotate scheduler extension | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate kubelet certificate | | | | | | |
| | | | restart ncp | NCP restart skipped: VDS setup | | | | | |
| | | | | detected. | | | | | |
| | | | rotate auth proxy certificate | | | | | | |
| | | | rotate management certificate | | | | | | |
| | | | rotate registry certificate | | | | | | |
| | | | rotate kubeadm admin | | | | | | |
| | | | certificate | | | | | | |
| | | | verify etcd health | | | | | | |
| | | +--------------------------------+------------------------------------+-------+ | | | |
| | | | | | |
| | +---------------------------------------------------------------------------------+----------------+ | |
| | | |
| 192.168.111.203 | +---------------------------------------------------------------------------------+----------------+ | |
| | | TASKS | OVERALL STATUS | | |
| | +---------------------------------------------------------------------------------+----------------+ | |
| | | +--------------------------------+------------------------------------+-------+ | ok | | |
| | | | TASK | RESULT | ERROR | | | | |
| | | +--------------------------------+------------------------------------+-------+ | | | |
| | | | backup certificates | /root/backups-16739895893751688144 | | | | | |
| | | | rotate etcd server certificate | | | | | | |
| | | | rotate api server etcd client | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate etcd peer certificate | | | | | | |
| | | | rotate etcd health check | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate api server certificate | | | | | | |
| | | | rotate kubelet client api | | | | | | |
| | | | server certificate | | | | | | |
| | | | rotate front proxy certificate | | | | | | |
| | | | rotate controller-manager | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate scheduler certificate | | | | | | |
| | | | rotate scheduler extension | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate kubelet certificate | | | | | | |
| | | | restart ncp | NCP restart skipped: VDS setup | | | | | |
| | | | | detected. | | | | | |
| | | | rotate auth proxy certificate | | | | | | |
| | | | rotate management certificate | | | | | | |
| | | | rotate registry certificate | | | | | | |
| | | | rotate kubeadm admin | | | | | | |
| | | | certificate | | | | | | |
| | | | verify etcd health | | | | | | |
| | | +--------------------------------+------------------------------------+-------+ | | | |
| | | | | | |
| | +---------------------------------------------------------------------------------+----------------+ | |
| | | |
| 192.168.111.201 | +--------------------------------------------------------------------------------+----------------+ | |
| | | TASKS | OVERALL STATUS | | |
| | +--------------------------------------------------------------------------------+----------------+ | |
| | | +--------------------------------+-----------------------------------+-------+ | ok | | |
| | | | TASK | RESULT | ERROR | | | | |
| | | +--------------------------------+-----------------------------------+-------+ | | | |
| | | | backup certificates | /root/backups-1673989589793637456 | | | | | |
| | | | rotate etcd server certificate | | | | | | |
| | | | rotate api server etcd client | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate etcd peer certificate | | | | | | |
| | | | rotate etcd health check | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate api server certificate | | | | | | |
| | | | rotate kubelet client api | | | | | | |
| | | | server certificate | | | | | | |
| | | | rotate front proxy certificate | | | | | | |
| | | | rotate controller-manager | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate scheduler certificate | | | | | | |
| | | | rotate scheduler extension | | | | | | |
| | | | certificate | | | | | | |
| | | | rotate kubelet certificate | | | | | | |
| | | | restart ncp | NCP restart skipped: NCP | | | | | |
| | | | | restart only occurs on the | | | | | |
| | | | | leader. | | | | | |
| | | | rotate auth proxy certificate | | | | | | |
| | | | rotate management certificate | | | | | | |
| | | | rotate registry certificate | | | | | | |
| | | | rotate kubeadm admin | | | | | | |
| | | | certificate | | | | | | |
| | | | verify etcd health | | | | | | |
| | | +--------------------------------+-----------------------------------+-------+ | | | |
| | | | | | |
| | +--------------------------------------------------------------------------------+----------------+ | |
| | | |
+------------------+------------------------------------------------------------------------------------------------------+-------+
+-----------------------------------------------------+----------------+
| TASKS | OVERALL STATUS |
+-----------------------------------------------------+----------------+
| +--------------------------------+--------+-------+ | |
| | TASK | RESULT | ERROR | | |
| +--------------------------------+--------+-------+ | |
| | rotate spherelet certificates | | | | |
| | on 192.168.111.56 (host-16) | | | | |
| | rotate spherelet certificates | | | | |
| | on 192.168.111.69 (host-22) | | | | |
| | rotate spherelet certificates | | | | |
| | on 192.168.111.93 (host-28) | | | | |
| +--------------------------------+--------+-------+ | |
| | |
+-----------------------------------------------------+----------------+
Main vSphere with Tanzu Cert Page: https://knowledge.broadcom.com/external/article?legacyId=89324.