Planned Activity Precheck Review Case for Supervisor Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Note: The Planned Activity Precheck Review outlined in this KB is available only to Advanced Support customers with a Support Account Manager (SAM) and/or Dedicated/Designated Support Engineer (DTSE/DSE). 

The Planned Activity Precheck Review is provided on a best effort basis and is intended to assist with identified issues and precheck failures. While Broadcom Support strives to highlight potential risks, this review is not exhaustive and may not identify all issues or environment-specific considerations. It is not a commitment to participate during the planned activity nor is it a certification or validation of a runbook. The customer remains responsible for validating their activity plan and ensuring their environment if fully prepared for the change. 

Prerequisites before opening a Planned Activity Precheck Review for Supervisor Cluster:

• Cases must be opened at least 10 business days in advance of the upgrade window
• Cases should be opened as Low-P4
• Supervisor Upgrade Pre-Checks Failure Guide must be run in advance and report any issues that cannot be resolved
• Check the disk space usage on all the Supervisor Cluster VM's before upgrading
  • The following KB can be used to SSH into Supervisor Cluster: Troubleshooting vSphere Supervisor Control Plane VMs
• Make sure all the certificates are valid on the Supervisor Cluster, refer to Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates
• Validate that the content library on vCenter is accessible and contains the subscribed or local libraries required for vSphere with Kubernetes
  • For VCF 9.X, ensure that the dedicated content library has Supervisor Cluster VM images that are accessible and contain the desired version to be upgraded to
• Validate if the NTP Synchronized. NTP should be configured on all ESXi hosts in the vSphere cluster that forms the Supervisor, as well as on the vCenter Server to prevent certificate and token time drift
  • esxcli system ntp get
  • ntpq -p
• Before 8.0u3, ensure that all workload clusters are on a compatible VKR version with the current vCenter version: vCenter and vSphere Kubernetes Releases Compatibility Matrix
• After 8.0u3, ensure that all workload clusters and the vSphere Kubernetes Service (VKS) version is compatible with the desired Supervisor cluster version: VKS, Supervisor and VKR Compatibility Matrix
  • If pre-upgrade checks warn regarding incompatible workload clusters post-upgrade of the Supervisor cluster, you must upgrade or delete the incompatible workload clusters before proceeding with the Supervisor cluster upgrade

VMware by Broadcom Support will:

• Ensure pre-check issues are resolved
• Validate the target version and advise if a newer one is available
• Review provided logs for known issues that may impact the upgrade, see the following KB for steps to retrieve a Workload/Supervisor Management Support Bundle: Gathering Logs for vSphere with Tanzu
• If a Planned Activity Precheck Review case is not raised with sufficient notice of 10 days, it may not be possible to offer log review and precheck resolution within the planned timeframe. 
• Once the review is complete, the Planned Activity Precheck Review case will be closed.
  • Two days prior to the upgrade, either you or your SAM/DTSE/DSE will open a maintenance case. This case will track the upgrade and can be uplifted to a Critical - P1 in the event of a production down outage during the upgrade. 
  • The next available support engineer will engage with you. Broadcom will not be able to provide dedicated support to be on standby during the activity. 

Please complete the following template when opening the Case: 

• Upgrade Date & Time?
• Current vCenter Server version?
• Current Supervisor Cluster version?
• Target Supervisor Cluster version?
• Product Interoperability checked?
• Have the release notes been reviewed for the target version?
• Do you have a file-based backup of the vCenter with the Supervisor Cluster included?
• Number of Supervisor Clusters being upgraded (if applicable)?
• Is the Supervisor Cluster managed by TMC?
• Deployment Type (NSX, VDS + HAProxy, VDS + AVI, VDS + FLB)?

Additional data to be collected using below commands:

• VDT tool output
• Use these commands to verify node and pod status, check overall cluster health, and validate etcd cluster member health and endpoint status.

Command	Purpose
kubectl get nodes -A	List all nodes in the cluster
kubectl get pods -A | grep -v Running	Check pods that are not in Running state
kubectl get --raw '/healthz?verbose'	Check overall Kubernetes cluster health
kubectl get deployment,daemonset -A	Check that pods match their expected replica count across namespaces
kubectl get pkgi -A	Check all Kapp-controlled package services health
etcdctl member list -w table	List etcd cluster members
etcdctl endpoint health --cluster -w table	Check health of etcd endpoints
etcdctl endpoint status --cluster -w table	View detailed etcd endpoint status

•  

Environment

• vSphere with Tanzu 8.x/9.x - Supervisor Cluster

Resolution

Open a Low - P4, Planned Activity Precheck Review case via Broadcom Support portal. 

The purpose of a Planned Activity Precheck Review case is to prepare in advance for the Supervisor Cluster upgrade.

If an issue is experienced during upgrade that causes a production down outage, Advanced Support customers should uplift the severity of the case to a Critical - P1.

This is not an exhaustive health check of the environment. If this is required, engage VMware by Broadcom Professional Services.

Additional Information

For further Planned Activity Precheck Review guidance, please see the Planned Activity Precheck Review - Broadcom Advanced Support KB which details the process and includes all VCF specific products this is offered for.