Error: "VCENTER upgrade failed to get service account due to service account does not exist" during vCenter patch via SDDC Manager
search cancel

Error: "VCENTER upgrade failed to get service account due to service account does not exist" during vCenter patch via SDDC Manager

book

Article ID: 452352

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

During a VMware Cloud Foundation (VCF) workload domain upgrade, the vCenter patch precheck may fail if the required service account is absent. This occurs when the SDDC Manager cannot retrieve or create the necessary service account for vCenter communication due to a Security Token Service (STS) certificate trust issue.

Symptoms:

  • vCenter patch via SDDC Manager fails with the error message: VCENTER <vCenter_FQDN> upgrade failed to get service account due to service account does not exist.

  • Attempt to create the service account by navigating to inventory in SDDC Manager > Workload Domains > Click on affected Workload Domain > Updates tab > scroll down to Available Configuration Updates > Apply all as per WLD Upgrade precheck error with message 'Service Account not found for communicating between SDDC manager and vCenter'  fails with the following error:

    Unable to create the service account on <vCenter_FQDN>. Unable to obtain Security Token Service from SSO <vCenter_FQDN>.

  • The /var/log/vmware/vcf/domainmanager/domainmanager.log on the SDDC Manager indicates a failure to fetch the STS certificate chain:

YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,####,####] [c.v.evo.sddc.common.util.SslUtil,dm-exec-####]  Getting certificate chain for the following certificates: [0]         Version: 3
         SerialNumber: ####
IssuerDN: OU=VMware Engineering, O=<vCenter_FQDN>, ST=California, C=US, DC=local, DC=vsphere, CN=CA
         Start Date: day month date hh:mm:ss UTC YYYY
         Final Date: day month date hh:mm:ss UTC yyyy
         SubjectDN: CN=ssoserverSign
         Public Key: RSA Public Key
YYYY-MM-DDThh:mm:ss  DEBUG [vcf_dm,####,####] [c.v.e.s.c.c.CertificateRetrieverService,dm-exec-####]  Certificate chain validity check against current PKIXParameters failed
Caused by: java.security.SignatureException: certificate does not verify with supplied key  
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,####,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-545]  [####] FAILED_TO_CREATE_SERVICE_ACCOUNT Unable to create the service account on <vCenter_FQDN>. Error: Unable to obtain Security Token Service from SSO '<vCenter_FQDN>'
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to create the service account on <vCenter_FQDN>. Error: Unable to obtain Security Token Service from SSO '<vCenter_FQDN>'   
Caused by: com.vmware.evo.sddc.common.services.psc.exception.PscException: Unable to obtain Security Token Service from SSO <vCenter_FQDN>
Caused by: java.lang.IllegalArgumentException: Expected one or more trusted certificates, but got null

Environment

  • VMware Cloud Foundation (VCF) 9.x

  • SDDC Manager 9.1

  • vCenter 9.1

Cause

A certificate chain validation failure happens during service account creation because the SDDC Manager truststore does not contain the signing certificate for the STS. This discrepancy occurs when the vCenter machine SSL and STS certificates are signed by completely different root certificates.

Resolution

  1. Identify the certificate signing the Machine SSL and STS certificates by comparing the Authority Key Identifier (AKID) of the vCenter machine SSL certificate against the Subject Key Identifier (SKID) of the STS Root CA certificate.

    1. Download and run the vCert tool on the vCenter as detailed in vCert - Scripted vCenter expired certificate replacement. 

    2. Select Option 2 (View Certificate Information).

    3. Select Option 1 to view the machine SSL certificate. Record the Authority Key Identifier (AKID).

    4. Select Option 8 to view the STS CA Certificate. Record the Subject Key Identifier (SKID) of STS CA Certificate.

    5. Verify whether the machine SSL certificate's Authority Key Identifier matches the STS CA Certificate's Subject Key Identifier.

  2. Inspect the SDDC Manager trusted certificate store.

    1. SSH to the SDDC Manager appliance as the vcf user, then run su to elevate privileges to root.

    2. KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

    3. keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY >/tmp/trusted.txt

    4. less /tmp/trusted.txt | grep -A 3 "SubjectKeyIdentifier"

    5. Confirm if the output contains a SubjectKeyIdentifier matching the STS CA Subject Key Identifier recorded in step 1.d.

  3. If the signing certificate of STS is missing in SDDC Manager, import it into SDDC Manager.

    1. SSH to the vCenter as root.

    2. Fetch the root certificate signing the STS certificate: /usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer

    3. Convert the root certificate on the vCenter into single-line PEM format: awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' /tmp/root.cer

    4. Log in to the SDDC Manager UI and navigate to Developer Center > API Explorer.

    5. Locate the Trusted Certificates API section and prepare a POST request.

    6. Provide the following parameters in the payload:

      1. certificate: Paste the single-line string generated in step 3.c.

      2. certificateUsageType: TRUSTED_FOR_OUTBOUND.

    7. Execute the POST call to insert the certificate.

  4. Re-run the VCF upgrade precheck.

Additional Information

  • To verify if the vCenter service account exists in the SDDC Manager, run the following commands:

      • TOKEN=$(curl -d '{"username" : "[email protected]", "password" : "<ADMIN_PASSWORD>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

      • curl -k -X GET -H "Authorization: Bearer $TOKEN" --insecure 'https://localhost/v1/system/credentials/service' | jq

  • Confirm the existence of the service account in vCenter by running: /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list

Powered by Wolken Software