NSX service account disconnected and password rotation failure in SDDC Manager
search cancel

NSX service account disconnected and password rotation failure in SDDC Manager

book

Article ID: 449684

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

This article describes the resolution for when the NSX service account shows as disconnected in the SDDC Manager Password Management dashboard, resulting in password rotation failures. This issue commonly impacts VMware Cloud Foundation (VCF) environments when credential operations fail.  The symptoms occur when SDDC Manager background authentication attempts trigger an account lockout loop on the NSX Manager.

Symptoms:

operationsmanager logs report:

<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] Error Message : The credentials were incorrect or the account specified has been locked., Error Token : <TOKEN_ID>
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] About to mark resource state as error...
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-17] Failure occurred in TEST_BEFORE_REMEDIATE stage
<Date && Time> DEBUG [vcf_om,0000000000000000,0000] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-17] Password operation failed for root in stage TEST_BEFORE_REMEDIATE

Retrieving the NSX Service account password, shows different secret values, at different intervals - Retrieve the service accounts credentials from SDDC Manager 
Use this alternative to view explicitly the Impacted NSX_Service_account
# curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | grep -i -B10 <NSX_Service_account>

Environment

VMware Cloud Foundation (VCF) 9.x, 9.1

NSX 9.X, 9.1 

Cause

The NSX service account is locked out on the NSX Manager. The SDDC Manager services continuously attempt to authenticate using outdated credentials, which immediately re-locks the account upon manual remediation attempts, creating a persistent lockout loop.

 

Resolution

  1. Perform a snapshot of the SDDC Manager before proceeding.
  2. Exempt the SDDC Manager from the NSX lockout policy by executing the following API command from the SDDC Manager:
    curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses += [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-
  3. Initiate the NSX service account password rotation from the SDDC Manager UI.
  4. Retrieve password via Retrieve the service accounts credentials from SDDC Manager
  5. Verify the service account remains connected after 5–10 minutes by Re-retrieving password.
  6. Rotate the remaining root, admin, or audit passwords as required for the scenario.
  7. Remove the SDDC Manager IP from the lockout immunity list by executing the following command from the SDDC Manager:
    curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses -= [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-


Additional Information

Collecting SDDC Manager and VMSP logs

If the issue persists, contact Broadcom Support: Contact Broadcom Support

Powered by Wolken Software