SDDC Manager Pre-check failing due to the vCenter in an ERROR status.

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• The pre-check on the SDDC Manager is failing for the MGMT vCenter with the below error:

  (1) One of the workflows ("Add Domain", "Password Manager") has put the VC in Error state. Check for current/previous workflow in the task panel in UI. Remediation messages for corresponding task can be found on Task Aggregator on UI. Follow the remediation steps for resolving the issue. (2) Check for failed VC/PSC upgrade, wait for a few minutes and retry.

• In some cases  we see  the below error is displayed on the SDDC Manager.

  Update not possible while vcsa is in failed state

• Error in /var/log/vmware/vcf/lcm/lcm.log

  ERROR [vcf_lcm,627######3d60,1528] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-10] Failed to get resource name for ESX_HOST with id 535b####-####-####-########5822
  ERROR [vcf_lcm,627######3d60,1528] [c.v.e.s.l.a.i.i.LogicalInventoryClient,http-nio-127.0.0.1-7400-exec-10] logical inventory - get ESXi host failed for ESXi host ID b4c4####-####-####-########2d59 org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"RESOURCE_NOT_FOUND_WITH_ID","arguments":["ResourceInventoryController","b4c4####-####-####-########2d59"],"message":"Resource ID: b4c4####-####-####-########2d59 not found in ResourceInventoryController","causes":[{"type":"com.vmware.evo.sddc.inventory.model.InventoryNotFoundException","message":"Resource ID: b4c4####-####-####-########2d59 not found in Esxi"}],"referenceToken":"R309LB"}"
  k.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)
  

• Error in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

  INFO  [common,529####a50f,154d] [c.v.e.s.i.r.a.c.ResourceInventoryController,http-nio-127.0.0.1-7100-exec-2938] getHosts(): id = b4c4####-####-####-########2d59
  INFO  [common,529####a50f,154d] [c.v.e.s.i.s.EsxiInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-2938] Get Esxi - b4c4####-####-####-########2d59
  ERROR [common,529####a50f,154d] [c.v.e.s.i.d.s.client.TypedClientImpl,http-nio-127.0.0.1-7100-exec-2938] Inventory Error, Resource ID: b4c4####-####-####-########2d59 not found in Esxi
  ERROR [common,529####a50f,154d] [c.v.e.s.i.s.EsxiInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-2938] Inventory Error, Resource ID: b4c4####-####-####-########2d59 not found in Esxi
  ERROR [common,529####a50f,154d] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-2938] [QLVA6P] RESOURCE_NOT_FOUND_WITH_ID Resource ID: b4c4####-####-####-########2d59 not found in ResourceInventoryController com.vmware.evo.sddc.inventory.model.InventoryNotFoundException: Resource ID: b4c4####-####-####-########2d59 not found in ResourceInventoryController
          at com.vmware.evo.sddc.inventory.rest.api.controller.ResourceInventoryController.getHosts(ResourceInventoryController.java:521)
          at jdk.internal.reflect.GeneratedMethodAccessor261.invoke(Unknown Source)

Cause

The vCenter column from the platform DB was in the ERROR status.

Resolution

1. SSH to SDDC Manager with vcf user and su to root
2. Find the vCenter in ERROR state in SDDC database

   psql -h localhost -U postgres -d platform -c "select id,vm_hostname,status from vcenter where status!='ACTIVE'"

   Sample output

                     id                  |     vm_hostname     | status
   --------------------------------------+---------------------+--------
    2730####-####-####-########d780 | vcsa02.example.com | ERROR
   (1 row)

3. Update the vCenter status to ACTIVE in SDDC Database

   psql -h localhost -U postgres -d platform -c "update vcenter set vm_hostname='ACTIVE' where id='<id from Step #2 >'"

4. Run the pre-check from SDDC Manager