Supervisor VM is failing to join cluster | Supervisor VM ovf mismatch

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

This Kb is to avoid k8s patch version mismatch between current supervisor and available ovf on vCenter. This issue could only occur in vCenter version 7U3d and below.

It is suggested not to perform any activity (except supervisor upgrade) on supervisor VMs immediately after vCenter upgrade. It may be possible that current k8s patch version won't be available on new vCenter.
Please check if your supervisor cluster is running on either of the new available k8s Patch version.

Also, If there is below banner (which would until you upgrade your cluster to latest available version) showing on Workload Management dashboard, Suggested to check, if your all wcp cluster is on available k8s version (mentioned in workload Management UI >> Updates tab), before doing any activity on particular wcp cluster.



Symptoms:

Newly recreated Supervisor VM is not able to join cluster even after it powered on successfully.

Issue observed after upgrade of vCenter to 7U3c from 7U2c in which supervisor k8s patch version v1.19.1 is not available on upgraded vCenter.

Running Supervisor Version -- 1.19.1
Available version in vCenter 7U3c -- 1.19.12, 1.20.8 and .1.21.0

We can see there is mismatch between supervisor vm ovf and available ovf rpm package version on vCenter terminal. In this scenario, if we recreate any of the VM then Supervisor cluster would go in degraded state.
Refer below command to check the same.

# /usr/lib/vmware-wcp/wcp-db-dump.py | grep master_vm_information -A 4
        "master_vm_information": {
          "vm-2009": "v1.19.1+vmware.2-vsc0.0.9-17882987",
          "vm-2010": "v1.19.1+vmware.2-vsc0.0.9-17882987",
          "vm-2011": "v1.19.1+vmware.2-vsc0.0.9-17882987"
        },

# rpm -qa VMware-wcpovf*
vmware-wcpovf-3.0.281-18735554.x86_64

Refer the last digits 17882987 and 18735554. Ideally, it should be same.

Environment

VMware vSphere 7.0 with Tanzu

Cause

This was occurred due to mismatch in Supervisor k8s patch version running on Supervisor VM and new wcp-ovf available on vCenter after vCenter upgrade.
 

Resolution

Fix is already been shipped in vCenter 7U3f to handle this patch version mismatch. 

Additional Information

Note :- Auto-upgrade only happens when there is unavailability of k8s minor version, not in patch version unavailability.

Impact/Risks:

New recreated supervisor VM won't able to join existing cluster.