Host commission from SDDC Manager fails at "Acquire SDDC Manager Host(s) locks"

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Host commission from SDDC Manager fails at 

  Acquire SDDC Manager Host(s) locks

   

• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

  WARN  [vcf_dm,95#########90,07fe] [c.v.v.c.s.i.ClusterManagerIsServiceImpl,http-nio-127.0.0.1-7200-exec-27]  Failed to get clusters information from resource aggregator. Cluster capacity information (CPU, Storage, Memory) won't be provided in the response.
  ERROR [vcf_dm,55#########9d,8f4c] [c.v.v.r.c.c.ResourceAggregatorServiceImpl,http-nio-127.0.0.1-7200-exec-43]  Failed to get clusters information from resource aggregator.
  com.vmware.cloud.foundation.rest.operationsmanager.internal.runtime.ApiException: Bad Gateway

   

• Error in /var/log/vmware/vcf/commonsvc/vcf-commonsvcs.log

  ERROR [common,66#####################ee,58cf] [c.v.e.s.l.s.impl.LockingServiceImpl,http-nio-127.0.0.1-7100-exec-311] No topology path found with type HOST, id ec01####-####-####-####-########06e2, and name esxi01.example.com
  ERROR [common,66#####################ee,58cf] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-311] [2####O] INVALID_RESOURCE Resource with type HOST, and ID ec01####-####-####-####-########06e2 or Name esxi01.example.com is not found.
  com.vmware.evo.sddc.common.core.error.InvalidInputException: Resource with type HOST, and ID ec01####-####-####-####-########06e2 or Name esxi01.example.com is not found.

   

• Error in /var/log/vmware/vcf/operationsmanager/oprationsmanager.log

  DEBUG [vcf_om,66###################00,5cb3] [c.v.v.p.v.u.ValidateCredentialsTranslationTaskExecutor,om-exec-1] Exception occurred during validate credentials translation task : Error creating bean with name 'liquibase': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
  org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'liquibase': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
          at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)

Cause

• Manual DB clean-up on the environment and some inventory resources were deleted. Therefore we see errors like missing cluster-domain association in the tables
• Unable to find one the host in the topology

Resolution

1. Take a snapshot of SDDC Manager VM
2. SSH to SDDC Manager with vcf user and elevate to root with su
3. Check for stale resource locks

   curl localhost/resource-locks | jq

   
   Sample output

   {
     "elements": [
       {
         "id": "0ca7####-####-####-####-########d8cb",
         "resourceType": "cluster",
         "resourceId": "029b####-####-####-####-########fbd7",
         "resourceName": "vi-cluster1",
         "operationId": "b53c####-####-####-####-########bbbe",
         "serviceId": "106c####-####-####-####-########5798"
       },
       {
         "id": "8a0e####-####-####-####-########bc0f",
         "resourceType": "system",
         "resourceId": "SYSTEM",
         "resourceName": "SYSTEM",
         "operationId": "b53c####-####-####-####-########bbbe",
         "serviceId": "106c####-####-####-####-########5798"
       },
       {
         "id": "42b2####-####-####-####-########2a78",
         "resourceType": "domain",
         "resourceId": "aea9####-####-####-####-########31f1",
         "resourceName": "nsxt-vi",
         "operationId": "b53c####-####-####-####-########bbbe",
         "serviceId": "106c####-####-####-####-########5798"
       }
     ]
   }

    

4. check if the operationId matches the workflow id of the failed workflow domain workflow
5. Delete the locks held by the operationid

   curl --location --request DELETE 'localhost/resource-locks' --header 'Content-Type: application/json' --data '{"operationId": "<operationId from Step#3>","serviceId": "<serviceId from Step#3>"}'

    

6. Retry host commission from SDDC Manager

If there are no stale resource locks after Step # 3 then restart commonsvcs service and retry the failed operation

systemctl restart commonsvcs