SDDC initiated NSXT upgrade fails immediately with 'Retrieving NSXT upgradable failed. Could not find the cluster resource details for the domain id {0}' error

Products

• VMware Cloud Foundation

Issue/Introduction

Within a minute of the NSXT upgrade being launched from SDDC it fails with the following error in the UI:

Retrieving NSXT upgradable failed. Could not find the cluster resource details for the domain id {0}

The log file, /var/log/vmware/vcf/lcm/lcm-debug.log has entries similar to:

YYYY-MM-DDT12:13:57.097+0000 ERROR [vcf_lcm,4ee157fb55b3484e,04e7] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,http-nio-127.0.0.1-7400-exec-9] Exception occurred while fetching the Hardware Support Info for the cluster <CLUSTER NAME> 93a0f0e4-379c-4b0d-9f83-#########
com.vmware.vapi.client.exception.BindingsException: Required operation parameter 'cluser' is missing
        at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:165)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:137)
        at com.vmware.esx.settings.clusters.enablement.SoftwareStub.get(SoftwareStub.java:47)
        at com.vmware.esx.settings.clusters.enablement.SoftwareStub.get(SoftwareStub.java:37)
        at com.vmware.vcf.vapi.vsphere.PersonalityImageManager.isEnabled(PersonalityImageManager.java:11

Environment

VMware Cloud Foundation 4.x

Cause

The cluster table in the SDDC database is missing 'source_id' values for one or more clusters

Resolution

1. Ensure a snapshot of the SDDC Manager VM is taken.
2. Note the amount of clusters being referenced in the logs:
   
   grep "Exception occurred while fetching the Hardware Support Info for the cluster" /var/log/vmware/vcf/lcm/lcm-debug.log | awk '{print $16}' | sort | uniq
3. Here we see it's 2 clusters.
   
   test-m02
test1-m02
   
   
4. Check the cluster table in the SDDC database:
   
   select source_id, name from cluster;
   
   Validate the source_id entry for the clusters from step 2. If the source_id is empty, it needs to be updated with the correct moid, "domain-c####"
   
   
5. To get the correct domain moid open the NSXT UI  --> Fabric  -->  Nodes, list the clusters and make note of the domain ID for both clusters.
6. Once identified, update the cluster table:
   
   update cluster set source_id='domain-c####' where name='test-m02';
   
   
   
7. Run the update query for each cluster identified.
8. Re-try the upgrade from the SDDC
   Note: In some cases, restarting the SDDC manager services may be required

Additional Information

The resolution requires database edits - a snapshot of the SDDC is necessary before implementing the resolution.