vSphere with Tanzu Guest Cluster nodes fail network assignment due to "opaque network with id not found"

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

• vSphere with Tanzu Guest Cluster node deployment will be stuck in Provisioning state
• New VM's are deployed for the Guest Cluster but never power on
• When viewing the network in the VM summary, the wrong network is selected
• Describing the VNETIF associated to the new VM object will show SuccessfulRealizedNSXResource
• VMOP logging on the Supervisor cluster will report that "opaque network with id ####### not found" for the VM object:

YYYY-MM-DDTHH:MM:SS.538786423Z stderr F E0713 10:51:12.538714       1 virtualmachine_controller.go:758] VirtualMachine "msg"="Provider failed to update VirtualMachine" "error"="opaque network with ID '<network-id>' not found" "name"="namespace-name/guest-cluster-name-l5trz-345bv987x-ltr9z"

• Searching for the reported network ID in NSX manager GUI will show the segment, Logical Ports, and Logical Switches created and present. 

Environment

VMware vSphere 7.0 with Tanzu
VMware vSphere 8.0 with Tanzu

Cause

The most common cause of this condition occurs when ESXi hosts have been added to the vSphere cluster that are NOT prepared for NSX-T.

The same error message related to "opaque network with ID not found" may also present if the Certificate between vCenter and NSX is modified or untrusted, or if there are multiple VDS's in the NSX OverlayTZ of which the particular Supervisor cluster is a member.

Resolution

There are two options for getting past this error.
1. Remove the unprepared ESXi hosts from the vSphere Cluster until they can be prepared for NSX-T,
2. Prepare all ESXi hosts participating in the vSphere Cluster for NSX-T operations. Reference documentation for steps detailing this process.