SDDC Manager Workload domain page fails to load with error - Failed to load VI domains general hardware detail information. Something went wrong. Please retry or contact the service provider and provide the reference token

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Failed to load VI domains general hardware detail information. Something went wrong. Please retry or contact the service provider and provide the reference token
• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

  ERROR [vcf_dm,xxxxxxxxxx67ec,d886] [c.v.v.v.c.ViManagerNsxtController,http-nio-127.0.0.1-7200-exec-6]  Received an exception while getting nsxtManagementCluster for domainId: Optional.empty
  
  org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"RESOURCE_NOT_FOUND_WITH_ID","arguments":["<component_name>","68b4####-####-####-####-########5cb5"],"message":"Resource ID not found in VxManager","referenceToken":"8###D6"}"

Environment

VMware Cloud Foundation

Cause

Manual deletion of resources like Cluster/domain/host/NSXT Edge from SDDC DB and resource ID reported in domainamanager.log is a stale entry in SDDC Manager database because of that manual deletion.

Resolution

1. Take Snapshot of SDDC Manager VM
2. SSH to SDDC Manager with vcf and su to root
3. Connect to platform database

   psql -h localhost -U postgres -d platform
   
   for VCF 5.1.1 - /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform

4. Check the vm_and_vm_type_and_domain table to determine the details of the resource id found in logs:

   select * from vm_and_vm_type_and_domain where vm_id='<Resource ID reported in domainmanager.log>';

   Sample

   select * from vm_and_vm_type_and_domain where vm_id='68b4####-####-####-####-########5cb5';
   

   id  |              domain_id               |                vm_id                 |         vm_type
   ---+--------------------------------------+--------------------------------------+-------------------------
   111 | 6c97####-####-####-####-########476e | 68b4####-####-####-####-########5cb5 | VXMANAGER 

   
    Here the resource id 68b4####-####-####-####-########5cb5 is of VxRail Manager

5. Check the vx_manager table for the resource id

   select id,vm_hostname,vm_name,cluster_id from vx_manager where id='68b4####-####-####-####-########5cb5';

6. Once confirmed it is a stale entry use DELETE command to remove the entry.

   delete from vx_manager where id='68b4####-####-####-####-########5cb5';
   delete from vm_and_vm_type_and_domain where vm_id='68b4####-####-####-####-########5cb5';

7. Reload SDDC Manager UI

 

Note: Based the output of the Step # 4, the table to check in Step # 5 will change.