VCF 9.1 Upgrade Precheck Failed: Service Account Not Found Between SDDC Manager and vCenter

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When performing a VMware Cloud Foundation (VCF) 9.1 upgrade precheck, the operation fails with the following error message:

  Error: Service Account not found for communication between SDDC Manager and vCenter

  
  
  

• Running the following command on the target vCenter Server via SSH reveals that the service account svc-xxxx-xxxxx-xxxxxx-xxx-xxxx is missing from the dedicated service account list, despite being visible as an active account under standard vCenter Single Sign-On (SSO) user management and SDDC Manager password management:

  /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list
  
  

Environment

• VMware Cloud Foundation (VCF) 9.1

• VMware Cloud Foundation (VCF) 9.x

Cause

During a prior remediation attempt for a disconnected credential state, the service account svc-xxxx-xxxxx-xxxxxx-xxx-xxxx was deleted and manually recreated as a standard SSO User account from the vCenter UI (Administrator > Single Sign On > Users and Groups) rather than as a dedicated SSO Service Account using dir-cli svcaccount create.

Resolution

1. Delete non-service account object from vCenter SSO:
   Log in to the vCenter UI and navigate to Administrator > Single Sign On > Users and Groups. Locate and delete the svc-xxxx-xxxxx-xxxxxx-xxx-xxxx user account.
   
   
2. Retrieve the stored password from the SDDC Manager API:
   
   
   • SSH into the SDDC Manager appliance as vcf and switch to root su -


   • Generate an API access token:
     TOKEN=$(curl -d '{"username" : "[email protected]", "password" : "<ADMIN_PASSWORD>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')


   • Extract the existing service account password stored in the SDDC Manager database:
     curl -k -X GET -H "Authorization: Bearer $TOKEN" --insecure 'https://localhost/v1/system/credentials/service' | jq
     
     
   • Locate and copy the Secret value assigned to svc-xxxx-xxxxx-xxxxxx-xxx-xxxx.
     
     
3. Recreate as a dedicated SSO Service Account:
   
   On the vCenter Server SSH session, create the account as a dedicated service account and update its password also called Secret to match the SDDC Manager database:
   
   
   • Create service account object
     /usr/lib/vmware-vmafd/bin/dir-cli svcaccount create --name svc-xxxx-xxxxx-xxxxxx-xxx-xxxx
     
     
   • Update the account password to match the retrieved SDDC Manager database secret:
     /usr/lib/vmware-vmafd/bin/dir-cli password change --account svc-xxxx-xxxxx-xxxxxx-xxx-xxxx
     (Enter the temporary generated password when prompted for current password, then paste the retrieved SDDC Manager password as the new password)
     
     
4. Grant required SSO group permissions:
   
   
   • Add the recreated service account to the necessary administrative SSO groups
     /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add svc-xxxx-xxxxx-xxxxxx-xxx-xxxx
/usr/lib/vmware-vmafd/bin/dir-cli group modify --name CAAdmins --add svc-xxxx-xxxxx-xxxxxx-xxx-xxxx
/usr/lib/vmware-vmafd/bin/dir-cli group modify --name SystemConfiguration.BashShellAdministrators --add svc-xxxx-xxxxx-xxxxxx-xxx-xxxx
     
     
5. Validate and Retry
   • Confirm the account is listed using  /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list command
     
     
   • Re-run the VCF upgrade precheck from SDDC Manager / VCF Operation

Additional Information

• Retrieve the service accounts credentials from SDDC Manager