vSphere with Tanzu with the NSX Load Balancer - Supervisor Cluster upgrade stuck at 3% with only one IP assigned to new SupervisorControlPlaneVM

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

• Supervisor Cluster upgrade is stuck at 3% on "Upgrade Namespaces cluster" task.

• New SupervisorControlPlaneVM is deployed with only one Management Network IP address assigned. No Workload Network IP is assigned and the VM has only one NIC for the Management Network.

• NCP logs show errors pointing towards a port ID from the ESXi host where the newly deployed supervisor VM has been placed.

From a Supervisor Control Plane node:

kubectl logs -n vmware-system-nsx -l component-nsx-ncp

 

[ncp GreenThread-38 E errorCode="NCP05014"] nsx_ujo.ncp.k8s.node_controller Failed to add node <node-name>: Unexpected error from backend manager (['<nsx-manager-name>:443', '<nsx-manager-name>:443', '<nsx-manager-name>:443', '<nsx-manager-name>:443']) for get_node_vif_or_tn_id: Failed to get segment port id or tn id for node <node-name> in cluster domain-<cluster-name>

• NSX Manager VMkernel adapters are missing in some of the ESXi hosts.

In vSphere Client > Inventory > ESXi host > Configure > Networking > VMkernel adapters

The host should have VMkernel adapters using the nsx-overlay and nsx-hyperbus TCP/IP stacks. These adapters will typically be named vmk10 and vmk50. 

 

Environment

VMware vSphere 7.0 with Tanzu

Cause

If the above symptoms are observed, it is likely that there is something wrong with the NSX configuration of the ESXi hosts.

Resolution

• Check the NSX configuration and make sure it is correctly configured for all ESXi hosts.
• Access the NSX Manager GUI and make sure the configuration correct according to the vSphere with Tanzu with NSX documentation.

Additional Information

Impact/Risks:

Note: as mentioned in the title of this KB, this is only applicable to Supervisor Clusters enabled with NSX networking stack.