Host Cluster Upgrade Initiated From VCF at NSX-T Fails with “No Value Present”

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
upgrade fails at host cluster as below

Pre-check for NSX-T audit fails with unknown exception for the corresponding workload domain

Environment

VMware Cloud Foundation 4.x

Cause

The codebase at SDDC currently uses hostname to compare the presence of a specific Compute Manager in sddc inventory. Since NSX-T allows setting either a FQDN or IP Address on its end it impacts upgrades

Lcm-debug.log

YYYY-MM-DDT18:02:14.993+0000 ERROR [vcf_lcm,b1c1be7a25bec661,ec64,auditId=0a4487f5-03b9-4113-a23d-70eafa7b0e2b,resourceType=NSX_T_MANAGER,resourceId= NSX-T manager name,name= NSX-T manager name] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-10] Error auditing NSX-T “NSX-T manager name” with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.NoSuchElementException: No value present
     at java.base/java.util.Optional.get(Optional.java:148)
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.markUpgradeAvailability(NsxtInventoryLoader.java:523)
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:83)
        ... 34 common frames omitted
YYYY-MM-DDT19:40:37.506+0000 ERROR [vcf_lcm,e887bd21b9ad2c68,c063,auditId=fc7e25d6-91c9-45a5-8a2a-7668f0e7928c,resourceType=NSX_T_MANAGER,resourceId= NSX-T manager name,name= NSX-T manager name] [c.v.e.s.l.p.i.n.NsxtInventoryLoader,Scheduled-3]Corresponding vCenter Mapping for Compute Manager #.#.#.# NOT FOUND in the VCF inventory
YYYY-MM-DDT19:40:39.040+0000 ERROR [vcf_lcm,e887bd21b9ad2c68,c063,auditId=fc7e25d6-91c9-45a5-8a2a-7668f0e7928c,resourceType=NSX_T_MANAGER,resourceId= NSX-T manager name name=NSX-T manager name] [c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,Scheduled-3] Error auditing NSX-T Cluster NSX-T manager name with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory

Resolution

Contact Broadcom Support for assistance implementing the resolution to this issue.

Additional Information

Workaround:

1. Verify the compute manager/vc name stored in SDDC db for the said wld vc
   1. Log into sddc with vcf user and switch to root
   2. Connect to db:
      • psql --host=localhost -U postgres -d platform
2. Run command
   • select * from vcenter;
3. Identify the name stored for the corresponding workload domain vCenter and have the same updated at NSX-T UI by following below steps:
   1. Go to NSX-T manager UI
   2. Go to Configuration->Fabric->Compute Managers.
   3. Edit the compute manager. Change IP of VCenter to FQDN of VCenter
   4. Perform rolling reboot of all NSX-T managers one after the other for syncing purposes.
4. Re-try the upgrade.