"Unable to deploy OVF Template Failed to attach VIF: The requested object : LogicalSwitch/xxxxxx-xxxx-xxxx-xxxx could not be found" VM provisioning getting failed in Guest Cluster.

Products

• VMware vSphere with Tanzu

Issue/Introduction

• Provisioning fails for VMs while editing the guest cluster. (Either Increase the worker nodes or change the VM Class)
  
  

• In Supervisor VM, vmop-controller-manager pod logs throws the below error msg. It is failing to deploy the image from content library due to logical switch not found.

Log Location: /var/log/pods/vmware-system-vmop_vmware-system-vmop-controller-manager-<Pod-ID>/manager/

EMMDD HH:MM:SS 1 virtualmachine_controller.go:263] VirtualMachine "msg"="Failed to reconcile VirtualMachine" "error"="deploy from content library failed for image \"ubuntu-20-xxxxxxxx\": deploy error: The operation failed due to An error occurred during host configuration: Failed to attach VIF: The requested object : LogicalSwitch/9xxxxxxx-xxxx-xxxx-xxxx-xxxxxx2 could not be found. Object identifiers are case sensitive.." controller/virtualmachine "msg"="Reconciler error" "error"="deploy from content library failed for image \"ubuntu-20\": deploy error: The operation failed due to An error occurred during host configuration: Failed to attach VIF: The requested object : LogicalSwitch/9xxxxxxx-xxxx-xxxx-xxxx-xxxxxx2 could not be found. Object identifiers are case sensitive.." "name"="<VM_Name>" "namespace"="<namespace_name>" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="VirtualMachine" EMMDD HH:MM:SS  1 virtualmachine_controller.go:352] VirtualMachine "msg"="Reconciling VirtualMachine" 

Environment

VMware vCenter Server

vSphere with Tanzu

Cause

Content Library looks for a logical switch while assigning a virtual Network Interface which is no longer there in any of the NSX-T segments but existed in vCenter's database attached to few of the distributed port groups.  VM's could not be assigned with any Network Interface and no IP address is being assigned to them as Content Library is trying to pick up a Logical Switch from the NSX which is already deleted.

Resolution

• Ensure that the logical switch no longer exists within the NSX-T and take valid Snapshot of vCenter Server (Powered Off Snapshots for all vCenter Servers in case of linked mode environment). 
  
  
• Open SSH to the vCenter Server and stop the 'vpxd' service
  
           service-control --stop vpxd


• Find out the list of distributed port groups having the affected logical switch attached to them and then delete the DVPG with the below commands
  
  
• Log into the vCenter's Postgres database: 

        /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres 

    select * from vpx_dvportgroup where logicalswitch_uuid like '%9xxxxxxx-xxxx-xxxx-xxxx-xxxxxx2%';

This will give the name of the Port Group to which logical switch is mapped to. 

Note : Logical Switch ID needs to be replaced as observed in the /var/log/pods/vmware-system-vmop_vmware-system-vmop-controller-manager-<Pod-ID>/manager/ logs.

• Copy the name of the Port Group from above and run the below command to confirm how many id's of the port group are present as the same port group can be attached to multiple VDS. 

    select * from vpx_entity where name = 'vnet-domain-cxx:xxxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx0-34';

• Take a backup of VCDB and ensure that no VM's are running on the above Port Group(s) and then proceed to delete them 

    delete from vpx_dvportgroup where logicalswitch_uuid like '%9xxxxxxx-xxxx-xxxx-xxxx-xxxxxx2%';

    delete from vpx_entity where name = 'vnet-domain-cxx:xxxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx0-34';

• Exit from the database and start vpxd service 

    service-control --start vpxd

• Once the affected Port Groups were removed from the vCenter database, VM's were being provisioned and network was assigned to them.