VCF Inventory Sync fails with error "Could not retrieve the VMCA root certificate for vCenter"

Products

• VMware SDDC Manager / VCF Installer
• VMware vCenter Server

Issue/Introduction

•  Inventory Sync task fails with "Could not retrieve the VMCA root certificate for vCenter".
• /var/log/vmware/vcf/domainmanager/domainmanager.log on the SDDC manager reports below error stack: 

  YYYY-MM-DDTHH:MM:SSZ.093+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]    File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_discovery/discover_domain.py", line 1237, in discover_certificates
  YYYY-MM-DDTHH:MM:SSZ.093+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]      vmca_pem: str = self.vcenter_rest_helper.get_vmca_root_certificate()
  YYYY-MM-DDTHH:MM:SSZ.093+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]    File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/api/vcenter/vcenter_rest_api_helper.py", line 110, in get_vmca_root_certificate
  YYYY-MM-DDTHH:MM:SSZ.093+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]      raise VcenterApiException(ErrorMessages.CANNOT_FIND_VMCA_ROOT_CERT_FORMAT.format(self.vcenter_address))
  YYYY-MM-DDTHH:MM:SSZ.093+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]  common.common_exceptions.VcenterApiException: Could not retrieve the VMCA root certificate for vCenter: example.vCenter.com
  YYYY-MM-DDTHH:MM:SSZ.096+0000 INFO  [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-26]  [YYYY-MM-DDTHH:MM:SSZ] [INFO] vcf_brownfield: Operation ESC[93msyncESC[00m completed on target: ESC[93mEXAMPLE-DOMAINESC[00m with status: ESC[93mFAILESC[00m in ESC[93m37.85ESC[00ms
  YYYY-MM-DDTHH:MM:SSZ.274+0000 ERROR [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-24]  Brownfield process failed with exit value 1 for operation sync with id #########-####-####-####-############
  YYYY-MM-DDTHH:MM:SSZ.274+0000 DEBUG [vcf_dm,#########-####-####-####-############,####] [c.v.v.v.s.b.u.BrownfieldImportProgressReportService,dm-exec-24]  Deserialized progress report of the task with id #########-####-####-####-############: {"process_id":"#########-####-####-####-############","status":"completed with error","progress":{},"total_steps":4,"errors":[{"error_code":"OPERATION_ISSUE","message":"Result status code from set datasource: 400","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"Result status code from vCenter VMCA root certificate retrieval: 503","severity":"*****"},{"error_code":"OPERATION_ISSUE","message":"Failed to sync domain EXAMPLE-DOMAIN","severity":"ERROR"}]}
  YYYY-MM-DDTHH:MM:SSZ.276+0000 ERROR [vcf_dm,#########-####-####-####-############,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-24]  [Q6GNK0] IMPORT_OPERATION_FAILED Import operation for sync with id #########-####-####-####-############ failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name EXAMPLE-DOMAIN --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/#########-####-####-####-############ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/#########-####-####-####-############ on the SDDC Manager appliance
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Import operation for sync with id #########-####-####-####-############ failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name EXAMPLE-DOMAIN --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/#########-####-####-####-############ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/#########-####-####-####-############ on the SDDC Manager appliance

Environment

• VMware Cloud Foundation 9.X

Cause

• This is a known race condition between Trustmanagement and STS in 9.0 because both services start up at the same time, yet Trustmanagement is dependent on STS.

Resolution

• The issue is addressed in VMware vCenter 9.0.2.0. Refer to the release notes: Log in to vCenter fails with an error for fetching identity providers
• As a temporary workaround, please retry the operation until it succeeds.