NSX svc account is locked after rotating password
search cancel

NSX svc account is locked after rotating password

book

Article ID: 456648

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • NSX svc account(svc-#####-####..) is disconnected from Password Management page on SDDC manager UI.

  • Network information from Network Settings page does not show up properly on SDDC manager UI.

  • From the operationsmanager log from SDDC, you can find the svc account auto rotate service was excuted and done without error.
    /var/log/vmware/vcf/operationsmanager/operationsmanager.log
[DATE/TIME] DEBUG [vcf_om,################################,####] [c.v.v.p.r.a.AutoRotatePasswordTrigger,pool-#-thread-#] Auto rotate service running...
...
[DATE/TIME] DEBUG [vcf_om,################,####] [c.v.v.p.helper.NsxtApiUtil,om-exec-##] Password updated for service user svc-####-###-######-###-##-####
[DATE/TIME] DEBUG [vcf_om,################,####] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-##] successfully updated password to css [NSX_FQDN]
[DATE/TIME] INFO  [vcf_om,################,####] [c.v.v.p.r.NewPasswordTransactionHandler,Thread-##] Password ROTATE operation ## SUCCEEDED
  • From NSX log side, password change was done but the account was locked after 1 consecutive failed login attempts.
    /var/log/auth.log.1
[DATE/TIME] NOTICE chpasswd ###### -  pam_unix(chpasswd:chauthtok): password changed for svc-###-###-###-###-###-####

/var/log/proxy/reverse-proxy.log
[DATE/TIME] WARNING NSX #### [nsx@### comp="nsx-manager" level="WARNING" logger="AuthenticationBlacklistService" subcomp="http" threadName="Processing request ########-####-####-####-############"] Accountsvc-###-###-###-###-###-####@IP_ADDRESS has been temporarily locked for 600 seconds after 1 consecutive failed login attempts.
  • The issue can happen not only executing auto rotate task but also manual password rotating for NSX manager svc account password.

Environment

SDDC Manager 9.1.X

NSX Manager 9.1.X

Cause

The NSX service account becomes locked due to a timing issue during asynchronous password rotation. 
Because the password rotation process is asynchronous, there is a small probability that previous password is used for a remote call at the exact moment NSX updates its password. 
Since the system is configured to allow only one authentication failure (api_max_auth_failures = 1), this single authentication mismatch immediately locks the service account.

Resolution

The Broadcom Engineering team is aware of this issue, and a fix is planned for an upcoming release.

Before applying patch for this issue, you can follow this steps as a workaround.

1. Connect SSH session to NSX manager with admin account and change max-auth-failures value for api like this from NSX CLI.

 (Check first and change it to default 5 )

 Check:  get auth-policy api max-auth-failures

 Change: set auth-policy api max-auth-failures 5

2. Exempt the SDDC Manager from the NSX lockout policy by executing the following API command from the SDDC Manager SSH session.

curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' | jq ".lockout_immune_addresses += [ \"$(hostname -i)\" ]" | curl -ks https://<NSX_VIP>/api/v1/cluster/api-service -u 'admin:<password>' -X PUT -H 'Content-Type:application/json' -d @-



Powered by Wolken Software