VCF Deployment Fails: Error Adding Service Account Users to vCenter Administrator Group

Products

• VCF Operations

Issue/Introduction

• During the deployment of VMware Cloud Foundation (VCF), the installation process fails at the stage "Add Service Account Users to Administrator Group in vCenter Server." 
  
  The failure message displayed is:
  

  Add Service Account Users to Administrator Group in vCenter Server

  Falled to add user svc-###sddc0#-<FQDN_OF_vCenteter>[email protected] to groups [Administrators, CAAdmins, SystemConfiguration.BashShellAdministrators] Reference Token: M###IB

  
  
  
• From /var/log/vmware/vcf/domainmanager/domainmanger.log show show a  STS certificate-related failure:
  
  
  YYYY-MM-DDThh:mm:ss.331+0000 DEBUG [vcf_dm,6930a9####f28eae88###e35cc,5381] [c.v.e.s.c.c.CertificateRetrieverService,dm-exec-22]  Certificate chain validity check against current PKIXParameters failed
  java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
          at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi_8.engineValidate(Unknown Source)
  :
  :
  YYYY-MM-DDThh:mm:ss.340+0000 DEBUG [vcf_dm,6930a9####f28eae88###e35cc,5381] [c.v.v.s.t.DynamicTrustManager,dm-exec-22]  Certificate chain O=VMware, L=Palo Alto, ST=California, C=US, CN=<FQDN_OF_VCENTER>, SerialNumber=167######931363 is valid
   
  YYYY-MM-DDThh:mm:ss.824+0000 ERROR [vcf_dm,6930a9####f28eae88###e35cc,5381] [c.v.e.s.common.util.SSOEntityService,dm-exec-22]  Exception while assigning groups to user: svc-###sddc01-<FQDN_OF_VCENTER>-3670
  com.vmware.evo.sddc.common.services.psc.exception.PscException: Unable to obtain Security Token Service from SSO '<FQDN_OF_VCENTER>'
          at com.vmware.evo.sddc.common.util.SSOEntityService.getSamlToken(SSOEntityService.java:320)
          at com.vmware.evo.sddc.common.util.SSOEntityService.createAdminClient(SSOEntityService.java:367)
          at com.vmware.evo.sddc.common.util.SSOEntityService.createAdminClient(SSOEntityService.java:327)
  :
  :
  Caused by: java.lang.IllegalArgumentException: Expected one or more trusted certificates, but got null
          at com.vmware.vim.sso.client.impl.X509TrustChainKeySelector.checkCtorArgsNotNull(X509TrustChainKeySelector.java:242)
          at com.vmware.vim.sso.client.impl.X509TrustChainKeySelector.<init>(X509TrustChainKeySelector.java:73)
  :
  :
  YYYY-MM-DDThh:mm:ss.828+0000 ERROR [vcf_dm,6930a9####f28eae88###e35cc,5381] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-22]  [MO###IB] FAILED_TO_ADD_USER_TO_GROUPS Failed to add user svc-###sddc01-<FQDN_OF_VCENTER>[email protected] to groups [Administrators, CAAdmins, SystemConfiguration.BashShellAdministrators]
   
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to add user svc-###sddc01-<FQDN_OF_VCENTER>[email protected] to groups [Administrators, CAAdmins, SystemConfiguration.BashShellAdministrators]
          at com.vmware.evo.sddc.vsphere.contract.AddUsersToGroups.execute(AddUsersToGroups.java:96)
          at com.vmware.evo.sddc.vsphere.contract.AddUsersToGroups.execute(AddUsersToGroups.java:30)
  
  
• Upon checking the status of the Security Token Service (STS) certificates, the root certificate is found to be missing:
  
  Checking STS Signing Certs & Signing Chains
-----------------------------------------------------------------
Checking TenantCredential-1:
   TenantCredential-1 signing certificate                 VALID
   TenantCredential-1 CA certificate                      MISSING
Checking TenantCredential-2:
   TenantCredential-2 signing certificate                 VALID
   TenantCredential-2 CA certificate                      MISSING

Environment

VMware Cloud Foundation (VCF) 9.0

Cause

• VCF's SDDC Manager relies on obtaining a Security Token from the vCenter's SSO service to perform administrative actions, such as adding the service account user to the required groups. A missing root certificate prevents the VCF installer from successfully validating the vCenter's certificate trust chain (Trust anchor for certification path not found), thus stopping the SSO authentication process.
  
  
• This failure typically leads to the exception: Unable to obtain Security Token Service from SSO.

Resolution

Note: Take a snapshot of the vCenter Server Appliance before attempting to renew or replace the STS Signing Certificate.

To resolve the issue renew/replace the STS Signing Certificate on the affected vCenter Server Appliance.

• Use below  to check the status of the STS certificate on the affected vCenter server
  
  "Checking Expiration of STS Certificate on vCenter Servers"
  
  
• Use the vCert tool on the vCenter Server Appliance to Renew/replace  the STS Certificate
  
  vCert - Scripted vCenter expired certificate replacement