Proactive Upgrade Case for Supervisor Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Note: The Proactive Upgrade process outlined in this KB is available only to Advanced Support customers with a Support Account Manager (SAM) and/or Dedicated Technical Support Engineer (DTSE). 

Prerequisites to be followed before opening Proactive Supervisor Cluster Upgrade Cases:

• Cases must be opened 10 business days in advance of the upgrade window
• Cases should be opened as Low-P4
• Supervisor Upgrade Pre-Checks Failure Guide must be run in advance and report any issues that cannot be resolved
• Check the disk space usage on all the Supervisor Cluster VM's before upgrading
  • The following KB can be used to SSH into Supervisor Cluster: Troubleshooting vSphere Supervisor Control Plane VMs
• Make sure all the certificates are valid on the Supervisor Cluster, refer to Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates
• Validate that the content library on vCenter is accessible and contains the subscribed or local libraries required for vSphere with Kubernetes
  • For VCF 9.X, ensure that the dedicated content library has Supervisor Cluster VM images that are accessible and contain the desired version to be upgraded to
• Validate if the NTP Synchronized. NTP should be configured on all ESXi hosts in the vSphere cluster that forms the Supervisor, as well as on the vCenter Server to prevent certificate and token time drift
  • esxcli system ntp get
  • ntpq -p
• Before 8.0u3, ensure that all workload clusters are on a compatible VKR version with the current vCenter version: vCenter and vSphere Kubernetes Releases Compatibility Matrix
• After 8.0u3, ensure that all workload clusters and the vSphere Kubernetes Service (VKS) version is compatible with the desired Supervisor cluster version: VKS, Supervisor and VKR Compatibility Matrix
  • If pre-upgrade checks warn regarding incompatible workload clusters post-upgrade of the Supervisor cluster, you must upgrade or delete the incompatible workload clusters before proceeding with the Supervisor cluster upgrade

VMware by Broadcom Support will:

• Ensure pre-check issues are resolved
• Validate the target version and advise if a newer one is available
• Review provided logs for known issues that may impact the upgrade, see the following KB for steps to retrieve a Workload/Supervisor Management Support Bundle: Gathering Logs for vSphere with Tanzu
• If a proactive Case is not raised with sufficient notice of 10 days, it will not be possible to offer log review and pre-check resolution within the planned time-frame

Please complete the following template when opening the Case: 

• Upgrade Date & Time?
• Current vCenter Server version?
• Current Supervisor Cluster version?
• Target Supervisor Cluster version?
• Product Interoperability checked?
• Have the release notes been reviewed for the target version?
• Do you have a file-based backup of the vCenter with the Supervisor Cluster included?
• Number of Supervisor Clusters being upgraded (if applicable)?
• Is the Supervisor Cluster managed by TMC?
• Deployment Type (NSX, VDS + HAProxy, VDS + AVI, VDS + FLB)?

Additional data to be collected using below commands:

• VDT tool output
• Use these commands to verify node and pod status, check overall cluster health, and validate etcd cluster member health and endpoint status.

Command	Purpose
kubectl get nodes -A	List all nodes in the cluster
kubectl get pods -A | grep -v Running	Check pods that are not in Running state
kubectl get --raw '/healthz?verbose'	Check overall Kubernetes cluster health
kubectl get deployment,daemonset -A	Check that pods match their expected replica count across namespaces
kubectl get pkgi -A	Check all Kapp-controlled package services health
etcdctl member list -w table	List etcd cluster members
etcdctl -cluster=true endpoint health -w table	Check health of etcd endpoints
etcdctl -cluster=true endpoint status -w table	View detailed etcd endpoint status

•  

Environment

vSphere with Tanzu 8.x/9.x - Supervisor Cluster

Resolution

Open a Low - P4, Upgrade Awareness Case via the Broadcom Support portal.

The purpose of a proactive upgrade case is to prepare in advance for the Supervisor Cluster upgrade.

This is not an exhaustive health check of the environment. If this is required, please engage VMware by Broadcom Professional Services.

If an issue is experienced during the upgrade, Advanced Support customers should uplift their maintenance case.