VCF - NSX-T Precheck and Upgrade attempt is failing on NSX-T Audit: "NSXT-Manager Audit failed with unknow exception"

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

VCF Pre-check fails on NSX-T manager with "NSX-T Manager Audit failed with unknown exception"

SDDC manager logs (/var/log/vmware/vcf/lcm/lcm-debug.log) report similar to below

[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx01 is an Edge Node or does not exist in VCF inventory
[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx02 is an Edge Node or does not exist in VCF inventory
[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx03 is an Edge Node or does not exist in VCF inventory
[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx04 is an Edge Node or does not exist in VCF inventory
[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx05 is an Edge Node or does not exist in VCF inventory
[vcf_lcm,d3cfaeb310f8bd07,d083,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-8] Transport Node nsxtesx06 is an Edge Node or does not exist in VCF inventory

Following exception traces can be seen in LCM debug log

YYYY-MM-DDTHH:MM:SS.XXXX ERROR [vcf_lcm,50ec64373cb1d6a1,decb,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local] 
[c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-9] Error auditing NSX-T Cluster nsxt.corp.local with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
    at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:85)
    at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtAuditImpl.doAudit(NsxtAuditImpl.java:68)
...
...
Caused by: java.util.NoSuchElementException: No value present
    at java.base/java.util.Optional.get(Optional.java:148)
    at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.markUpgradeAvailability(NsxtInventoryLoader.java:523)
    at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:83) 
YYYY-MM-DDTHH:MM:SS.XXXX INFO [vcf_lcm,50ec64373cb1d6a1,decb,auditId=xxxxxxxx-xxxx-xxxx-xxxxxxxxx,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local,name=nsxt.corp.local]
[c.v.e.s.lcm.audit.NsxtAuditService,Scheduled-9] NSX-T Audit returned FAILED for resource type NSX_T_MANAGER with name nsxt.corp.local

Environment

VMware Cloud Foundation 4.x

Cause

SDDC Manager queries the NSX-T Inventory by name. If the names of the transport nodes do not match the names of the ESXi hosts in the SDDC Manager inventory, it will mark the transport node as 'does not exist in VCF inventory'. If this happens for all the hosts in a cluster, the audit workflow fails with the error: Failed to load NSX-T Cluster from the Inventory - No value present.

Resolution

Follow the steps provided below to resolve this issue:

1. SSH into the SDDC Manager with the vcf user and then su root.

2. Get the names of all hosts failing with this error:

less lcm-debug.log | grep -i 'does not exist in VCF inventory' | awk '{print $7}' | sort -d | uniq

(Note: In the output above, consider only the ESXi hosts. Edge Transport nodes can be ignored)

3. Confirm the names of the hosts in the SDDC Manager.
This can be done via the SDDC Manager UI > Workload Domain > Hosts

4. Login to the NSX-T UI > System > Fabric > Nodes > Transport Nodes
Select the VC to which the host(s) in question are in Managed By.

5. Edit the Transport Host to change the display name from the shortname to the FQDN (as we see in the SDDC Manager)

6. Once all the hosts (from step 2) are renamed, initiate a rolling reboot of all the NSX-T manager nodes.
Note: Ensure the nodes are rebooted one by one and the status of the node is Green in the System > Appliances tab before starting the reboot of the next.

7. Run the PreCheck from the SDDC Manager UI again - we should no longer see an NSX-T Audit failure with this error.