"NSX-T Manager Audit failed with unknown exception" failure for NSX-T Manager during VCF Upgrade precheck

Products

• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• VCF Pre-check fails on NSX-T manager with:

  NSX-T Manager Audit failed with unknown exception

• /var/log/vmware/vcf/lcm/lcm.log in SDDC manager reports similar to below:

  ####-##-##T##:##:##.###Z ERROR [vcf_lcm,0c4fe0dfc82a4479,60c3,auditId=########-####-####-####-##########92,resourceType=NSX_T_MANAGER,resourceId=abc.example.com] [c.v.e.s.l.p.impl.nsxt.NsxtAud
  itImpl,pool-6-thread-9] Error auditing NSX-T Cluster labc.domain.com with exception {}
  com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
          at  com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:87)
          at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtAuditImpl.doAudit(NsxtAuditImpl.java:68)
          at com.vmware.evo.sddc.lcm.audit.NsxtAuditService.doAudit(NsxtAuditService.java:120)
          at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler.auditNsxtInventory(NsxtInventoryAuditScheduler.java:84)
          at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler$$FastClassBySpringCGLIB$$2ae06d0f.invoke(<generated>)
          at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
  Caused by: java.lang.RuntimeException: NSX-T Upgrade Coordinator could not be fetched
          at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getNsxtUcConnection(InventoryClientImpl.java:1706)
          at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl$$FastClassBySpringCGLIB$$82f85f63.invoke(<generated>)
          at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
          at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
          at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl$$EnhancerBySpringCGLIB$$985e7950.getNsxtUcConnection(<generated>)
          at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtComponents(NsxtInventoryLoader.java:351)
          at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:83)
          ... 35 common frames omitted

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary for the specific environment.

Environment

• VMware NSX-T Data Center
• VMware Cloud Foundation

Resolution

To resolve this issue:

1. Confirm that the default response to the following API command returns a fully-qualified domain name (FQDN):

   root@sddc-manager [ /home/vcf ]# curl -k -u admin -X GET -H "Content-Type:application/json" https://<nsxt-manager-ip>/api/v1/node/services/install-upgrade | json_pp
   {
      "service_name" : "install-upgrade",
      "_self" : {
         "rel" : "self",
         "href" : "/node/services/install-upgrade"
      },
      "_schema" : "NodeInstallUpgradeServiceProperties",
      "service_properties" : {
         "enabled" : true,
         "enabled_on" : "example.domain.tld"   ← Here it returned an FQDN instead of an IP address, as expected.
      }
   }

2. If the output for enabled_on in the actual environment matches the above and contains an FQDN, follow below steps:
   1. Open an SSH connection with the node that should be set as an orchestrator node.
   2. Run the following command:

      # set repository-ip ##.##.##.##

   3. Re-run above API command, and confirm that it is returning the following expected response:

      vcf@sddc-manager [ ~ ]$ curl -k -u admin -X GET -H "Content-Type:application/json" https://<nsxt-manager-ip>/api/v1/node/services/install-upgrade | json_pp
       {
         "service_properties" : {
            "enabled" : true,
            "enabled_on" : "##.##.##.##"   ← Expected to be the the node where the upgrade service was enabled
         },
         "_schema" : "NodeInstallUpgradeServiceProperties",
         "_self" : {
            "href" : "/node/services/install-upgrade",
            "rel" : "self"
         },
         "service_name" : "install-upgrade"
      }