Aria Automation builds failing when trying to contact NSX, error "SecurityException: : : Getting virtual machine on NSX-T policy endpoint with name: [<Machine name>], external id: [...].: Max retries exceeded retry duration 600000 milliseconds"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware NSX

Issue/Introduction

• VM Deployments in Aria Automation are failing with error:
  SecurityException: : : Getting virtual machine on NSX-T policy endpoint with name: [<Machine name>], external id: [...].: Max retries exceeded retry duration 600000 milliseconds
• the VM can be seen in vSphere as powered on, but the VM did not receive the IP address
• the VM can not be found in the NSX inventory
• reviewing the provisioning service logs (/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log), the following can be observed:
   

  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-queries-13' user='' org='' trace='' parent='' span=''] c.v.m.a.nsxt.NsxTResourceTaggingService.checkApiType:254 - Using ApiType [policy] for tagging resource [VMNAME] on endpoint with name [###]
  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-queries-4' user='' org='' trace='' parent='' span=''] c.v.m.adapter.nsxt.NsxTPolicyApiClient.callApiAsync:1416 - Retrieving machines [VMNAME]

  The process then tries to retrieve the VM from the NSX API, it is marked as SUCCESS as the API call completed, however as the VM is not in the result of the API call the below logging repeats for 10 minutes:

  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-queries-4' user='' org='' trace='' parent='' span=''] c.v.m.a.nsxt.NsxTResourceTaggingService.log:416 - [8282/provisioning/nsxt/resource-tagging-adapter] Getting virtual machine on NSX-T policy endpoint with name: [VMNAME], external id: [2#######-####-####-####-############].: SUCCESS
  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-queries-4' user='' org='' trace='' parent='' span=''] c.v.m.a.nsxt.NsxTResourceTaggingService.log:416 - [8282/provisioning/nsxt/resource-tagging-adapter] Getting virtual machine on NSX-T policy endpoint with name: [VMNAME], external id: [2#######-####-####-####-############].: doRetry in (ms): 242

  After 10 minutes the process timed out and the deployment failed:

  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-service-scheduled-2' user='' org='' trace='' parent='' span=''] c.v.m.adapter.nsxt.NsxTPolicyApiClient.callApiAsync:1416 - Retrieving machines [VMNAME]
  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-service-scheduled-2' user='' org='' trace='' parent='' span=''] c.v.m.a.nsxt.NsxTResourceTaggingService.log:416 - [8282/provisioning/nsxt/resource-tagging-adapter] Getting virtual machine on NSX-T policy endpoint with name: [VMNAME], external id: [2#######-####-####-####-############].: SUCCESS
  INFO provisioning [host='provisioning-service-app-##########-#####' thread='xn-service-scheduled-2' user='' org='' trace='' parent='' span=''] c.v.m.a.nsxt.NsxTResourceTaggingService.log:416 - [8282/provisioning/nsxt/resource-tagging-adapter] Updating tags on virtual machine [VMNAME] on NSX-T policy endpoint.: FAILED with java.util.concurrent.CompletionException: com.vmware.xenon.common.LocalizableValidationException: Getting virtual machine on NSX-T policy endpoint with name: [VMNAME], external id: [2#######-####-####-####-############].: Max retries exceeded retry duration 600054 milliseconds.

 

Environment

Aria Automation 8.18.x

NSX 4.x

Cause

Aria Automation tried to locate the VM within the NSX inventory in order to create the NSX VM tag, however as the VM was not found within 10 minutes the process timed out.

Resolution

As the issue occurs on NSX side, it needs to be further investigated to why the VM is not showing within the NSX inventory.

A possible cause is documented with this article: "Error: 'SecurityException Getting virtual machine on NSX-T policy endpoint" while deploying a virtual machine with Aria Automation

If required please contact Broadcom Support to assist to raise a Support Ticket for NSX.