NSX service account disconnected and password rotation failure in SDDC Manager
search cancel

NSX service account disconnected and password rotation failure in SDDC Manager

book

Article ID: 449684

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

This article describes the resolution for when the NSX service account shows as disconnected in the SDDC Manager Password Management dashboard, resulting in password rotation failures. This issue commonly impacts VMware Cloud Foundation (VCF) environments when credential operations fail.  The symptoms occur when SDDC Manager background authentication attempts trigger an account lockout loop on the NSX Manager.

Symptoms:

  • NSX service account shows as "Disconnected" in SDDC Manager Password Management UI.

  • Password rotation for root, admin, or audit accounts fails.

  • Credential operations fail on NSX-T Components in SDDC Manager does not help  

  • Attempting to rotate the NSX service account password shows expired admin password and sometimes circular credential lockout.

operationsmanager logs report:

<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] Error Message : The credentials were incorrect or the account specified has been locked., Error Token : <TOKEN_ID>
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] About to mark resource state as error...
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] Failure occurred in TEST_BEFORE_REMEDIATE stage
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-17] Password operation failed for root in stage TEST_BEFORE_REMEDIATE

Retrieving the NSX Service account password, shows different secret values, at different intervals - Retrieve the service accounts credentials from SDDC Manager 
Use this alternative to view explicitly the Impacted NSX_Service_account
# curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | grep -i -B10 <NSX_Service_account>

Environment

VMware Cloud Foundation (VCF) 9.x, 9.1

NSX 9.X, 9.1 

Cause

The NSX service account is locked out on the NSX Manager. The SDDC Manager services continuously attempt to authenticate using outdated credentials, which immediately re-locks the account upon manual remediation attempts, creating a persistent lockout loop.

 

Resolution

  1. Perform a snapshot of the SDDC Manager before proceeding.
  2. Exempt the SDDC Manager from the NSX lockout policy by executing the following API command from the SDDC Manager:
    curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses += [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @- 
  3. Initiate the NSX service account password rotation from the SDDC Manager UI.
  4. Retrieve password via Retrieve the service accounts credentials from SDDC Manager
  5. Verify the service account remains connected after 5–10 minutes by Re-retrieving password.
  6. Rotate the remaining root, admin, or audit passwords as required for the scenario.
  7. Remove the SDDC Manager IP from the lockout immunity list by executing the following command from the SDDC Manager:
    curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses -= [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-

If the above fails to resolve your issue, consider the following below:

  1. Validated and updated the expired NSX admin password to restore administrative operations. We also checked and updated other essential credentials (root/audit) that were found to be expired.
  2. Temporarily disabled the NSX API lockout policy (set auth-policy api max-auth-failures 0) to allow for service account password operations without triggering security blocks.
  3. Identified the specific service account userid and performed a manual password reset via the NSX API (/api/v1/node/users/<id>?action=reset_password).
    curl https://localhost/api/v1/node/users?filter=service -u admin -kv
    
    curl https://localhost/api/v1/node/users/11000?action=reset_password -X POST -H "Content-Type: application/json" -d '{"password": ""}' -kv
    
    curl https://localhost/api/v1/reverse-proxy/password-update -X POST -H "Content-Type: application/json" -d '{"username": "", "old_password": "", "new_password": ""}'
    
    restart service http
    restart service auth
    
    
  4. Sync the password in SDDC-M using the below command:
    curl -ikX PUT  -d '<new password>' -H "Authorization: Bearer $t" -H "Content-Type: application/json" --insecure https://localhost/v1/system/credentials/{id}
    
    
  5. Remediated the credentials in the SDDC Manager Password Management dashboard and verified synchronization.

Additional Information

Collecting SDDC Manager and VMSP logs

If the issue persists, contact Broadcom Support: Contact Broadcom Support