Virtual Machine deployment via vRA fails post NSX restore with error "Could not find network with logical switch id <id> among target <domain-id> networks after 12 retries"

Products

• VMware vCenter Server

Issue/Introduction

• Due to environmental storage issues, NSX Manager nodes had to be restored.
• Post the restore, Virtual Machine deployment via vRA fails with the following error:
  
  Provisioning operation failed. Could not find network with logical switch id <id> among target domain-c#### networks after 12 retries.


• vRA provisioning logs report the following error:
  
  /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log

YYYY-MM-DDTHH:MM:SS Z WARN provisioning [host='provisioning-service-app-#####-#####' thread='vsphere-io-47' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterInstanceService.cleanupInstanceResource:778 - Entering cleaning up instance [#####] resource due to a throwable: [java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Could not find network with logical switch id '<id>' among target 'domain-c#####' networks after 12 retries.
                at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.createInstanceAsync(VSphereAdapterInstanceService.java:710)
                at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.lambda$handleCreateInstanceAsync$17(VSphereAdapterInstanceService.java:278)
                at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
                at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
                at com.vmware.photon.controller.model.adapters.vsphere.executor.ContextWrapperUtil.lambda$wrapWithOperationContext$0(ContextWrapperUtil.java:24)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at java.base/java.lang.Thread.run(Thread.java:840)
        Caused by: java.lang.IllegalArgumentException: Could not find network with logical switch id '<id>' among target 'domain-c####' networks after 12 retries.
                at com.vmware.photon.controller.model.adapters.vsphere.InstanceClient.findNsxNetworkForPlacement(InstanceClient.java:4562)
                at com.vmware.photon.controller.model.adapters.vsphere.InstanceClient.networkMoref(InstanceClient.java:1372)
                at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.getNetworkSpec(VSphereAdapterInstanceService.java:755)
                at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.createInstanceAsync(VSphereAdapterInstanceService.java:682)
                ... 7 more

Environment

vCenter Server 8.x
VMware Aria Automation 8.18.x

Cause

VMware Aria Automation expects that the Managed Object ID (MOID) and the portGroupKey value for Distributed Virtual Port Groups (DVPGs) is always identical. However, this framework fails in environments where a network restore has been performed in the vCenter.  When the option "Preserve original distributed switch and port group identifiers" is selected during the restore process, the portGroupKey value remains unchanged, but the MOID is regenerated. This results in a difference between the MOID and the portGroupKey value, preventing vRA from querying the correct Distributed Virtual PortGroup(DVPG) and leading to deployment failures.

To verify the difference in the MOID and the portGroupKey value, follow the below steps:

1. Access vCenter MOB: https://VC_FQDN/mob
2. Navigate: content-> rootfolder-> childEntity (pick the associated Datacenter)-> networkFolder-> childEntity (pick the DVPG facing the issue)
3. Review the key value and MOID in the URL for the DVPG. 
   
    
   
   

Note: A difference between the MOID and portGroupKey value is expected after a network restore in vCenter.

Resolution

This is a known issue with VMware Aria Automation and Engineering is working on a fix to be released in a future version.

Reach out to Broadcom Support under the VMware Aria Automation Product category for further assistance.