NSX service account is disconnected in SDDC manager Password Management

Products

• VMware SDDC Manager / VCF Installer
• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

• The NSX svc-vcf-nsxmgr-#### service account shows disconnected in the SDDC manager > Password management
  

OR

• The NSX svc-vcf-nsxmgr-#### service account shows blank or -- along with audit, root and admin also appearing to be blank or -- 
• Remediation for the NSX svc-vcf-nsxmgr-#### service account password is failing with the error : 
  "Cannot complete login due to an incorrect user name or password." 
  
  
  
• /var/log/vmware/vcf/operationsmanager/operationsmanager.log (on SDDC Manager) :
  YYYY-MM-DDTHH:MMM:SS ERROR [vcf_om,################################,8bb1] [c.v.v.p.u.c.NsxtManagerApiChanger,om-exec-4] Exception occurred while changing password for user <nsx_service_account>
YYYY-MM-DDTHH:MMM:SS ERROR [vcf_om,################################,8bb1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-4] The credentials were incorrect or the account specified has been locked.
YYYY-MM-DDTHH:MMM:SS ERROR [vcf_om,################################,8bb1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-4] The credentials were incorrect or the account specified has been locked.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: The credentials were incorrect or the account specified has been locked.
Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "{<EOL>  "error_code" : 403,<EOL>  "module_name" : "common-services",<EOL>  "error_message" : "The credentials were incorrect or the account specified has been locked."<EOL>}"
YYYY-MM-DDTHH:MMM:SS DEBUG [vcf_om,################################,8bb1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-4] Error Message : The credentials were incorrect or the account specified has been locked., Error Token : 
YYYY-MM-DDTHH:MMM:SS DEBUG [vcf_om,################################,8bb1] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-4] Password operation failed for <nsx_service_account> in stage UPDATE
• If an upgrade readiness pre-check is performed from SDDC, it will fail with: Cannot connect to <NSX_Manager> 
  
  

Environment

VMware Cloud Foundation 9.x 
VMware SDDC Manager 9.x
NSX 9.x 

Cause

The credentials stored in the SDDC Manager database no longer match the password active on the NSX Manager, SDDC Manager fails to authenticate. When SDDC Manager attempts to perform background operations or password remediation, the authentication failure returns an error ("The credentials were incorrect or the account specified has been locked"), causing the service account to display as Disconnected in the SDDC Manager UI.

Resolution

Perform the password remediation for the service account :

On NSX Manager : 

1. Login to NSX Manager via SSH using root credentials. Will likely need to enable root access using KB:Enable ssh root access for NSX appliances
2. To stop the service, run the command : 
   /etc/init.d/nsx-mp-api-server stop
3. Clear password history:
   echo "" >/etc/security/opasswd
4. Set the password to match what is present in SDDC DB. Use KB:Retrieve the service accounts credentials from SDDC Manager to pull the service acct password from SDDC Manager. 
   passwd <serviceacctUserName>
5. Run the command to create the file to reset the password: 
touch /var/vmware/nsx/reset_cluster_credentials
6. To start the service, run the command :
   /etc/init.d/nsx-mp-api-server start

On SDDC Manager : 

Remediate the password for the service account, with the password that was set and matches SDDC Manager. See: Remediate Passwords for VMware Cloud Foundation Components.

Additional Information

SDDC Manager Password Remediation fails due to 127-character UI limit