Password remediation for Avi service account fails in SDDC Manager due to duplicate credential entries

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Password remediation for the Avi service account ([email protected]) fails when triggered from SDDC Manager.

• The remediation task in SDDC Manager fails repeatedly.

• Manually updating the password in all vCenter Servers and retrying the remediation from SDDC Manager does not resolve the issue.

• Filtering by "vCenter" in Password Management reveals two or more identical service account entries for the same vCenter. There will be two or more svc users present in the Password management. If we navigate to password management > Filter with vCenter, two or more svc users are visible for vCenter with the NSX ALB.

• Running this command gives two or more identical users from SDDC DB: psql -h localhost -U postgres -d platform -c "select id, entitytype, username, servicetype, serviceid from credential where username like '%[email protected]%';"

  psql -h localhost -U postgres -d platform -c "select id, entitytype, username, servicetype, serviceid from credential where username like '%[email protected]%';"
                    id                  | entitytype |            username             | servicetype  |              serviceid
  --------------------------------------+------------+---------------------------------+--------------+--------------------------------------
   e######4-0##5-4##c-9e##-ee##########d | VCENTER    | [email protected]   | NSX_ALB      | 0######e-8###-4###-81##-b35##########1a
   e######3-0##4-4##6-b6##-87##########4 | VCENTER    | [email protected]   | NSX_ALB      | 0######e-8###-4###-81##-b3############a
  (6 rows)

Environment

VMware Cloud Foundation 5.x

Cause

This issue occurs if a previous attempt to deploy the NSX Advanced Load Balancer (Avi) failed. Although the deployment failed, a service account entry was created in the SDDC Manager database and the vCenter Server. These stale records are not always purged automatically, leading to duplicate entries that conflict with the password management workflow.

Resolution

Validate the duplicate svc user by reviewing the last modified time. Using the unique ID from GET /v1/credentials, remove it from the SDDC manager DB.

1. Take a snapshot of the SDDC manager VM.
2. SSH to the SDDC Manager as vcf and switch to root.
3. Generate a Token

   TOKEN=$(curl -H 'Content-Type:application/json' https://localhost/v1/tokens -d '{"username" : "<sso_user>","password":"<sso_password>"}' -k | jq -r '.accessToken')

4. Fetch credentials for resource type vCenter

   curl -k -X GET "https://<sddc_manager_fqdn>/v1/credentials?resourceType=VCENTER" -H "Authorization: Bearer $TOKEN" | jq '.elements[] | {id, username, modificationTimestamp}'

   
   Sample output

   {
     "id": "c######d-9##6-4##3-8##2-0########9",
     "username": "root",
     "modificationTimestamp": "2025-01-08T18:43:17.136Z"
   }
   {
     "id": "e######4-0##5-4##c-9e##-ee##########d", ===============> Stale entry
     "username": "[email protected]",
     "modificationTimestamp": "2025-01-08T18:43:17.320Z" ============> Older timestamp
   }
   {
     "id": "e######3-0##4-4##6-b6##-87##########4",
     "username": "[email protected]",
     "modificationTimestamp": "2026-04-15T19:03:12.020Z"
   }

   Note: username "[email protected]" has 2 entries, however, the modification time for id "e######4-0##5-4##c-9e##-ee##########d" has an older timestamp which confirms it is a stale entry.

5. Delete the stale credential

   curl -k -X DELETE "https://<sddc_manager_fqdn>/v1/system/credentials/<stale_id_from Step#4>" -H "Authorization: Bearer $TOKEN"

Note: Once the duplicate svc-xxx-avi user is removed, ensure to validate the functionalities are working as expected from the NSX load balancer. Removing the incorrect user can cause inconsistencies for the services required to be performed by the svc user.