Password rotation for NSX-T manager service account on SDDC fails in VCF 9.x - Failed to get NSX user details.

Products

• VMware SDDC Manager

Issue/Introduction

• Unable to rotate the svc account for NSX managers. 
  
  
• There are 2 svc accounts. The rotation fails for both these accounts.

svc-sddc_fqdn-nsxt_fqdn-#### , Resource Type NSXT
[email protected], Resource Type vCenter server 

• Password management operations - remediate/update/rotate on either of the account fails with the below error -

Failed to get NSX user details. Cause : The credentials were incorrect or the account specified has been locked.

• /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

yyyy-mm-ddThh:mm:ss INFO [vcf_om] [c.v.v.p.helper.NsxtApiUtil,om-exec-26] URI created is : https://nsx_manager_FQDN/api/v1/node/users/11000?action=reset_password
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.helper.NsxtApiUtil,om-exec-26] Auth Header creation successful !

yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.helper.NsxtApiUtil,om-exec-26] URI created is : https://nsx_manager_FQDN/api/v1/reverse-proxy/password-update
yyyy-mm-ddThh:mm:ss INFO [vcf_om] [o.b.jsse.provider.ProvTlsClient,om-exec-26] [client #29693 @2800a130] opening connection to nsx_manager_FQDN:443
yyyy-mm-ddThh:mm:ss INFO [vcf_om] [o.b.jsse.provider.ProvTlsClient,om-exec-26] [client #29693 @2800a130] established connection with nsx_manager_FQDN:443

yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.c.a.i.VcfApiTelemetryInterceptor,http-nio-127.0.0.1-7300-exec-7] is public API : true
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.s.PasswordUpdateHistoryService,http-nio-127.0.0.1-7300-exec-7] Getting password update history with page 1, page size: 2147483647, exclude passwords: true
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.c.a.i.VcfApiTelemetryInterceptor,http-nio-127.0.0.1-7300-exec-6] is public API : true
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.s.PasswordUpdateHistoryService,http-nio-127.0.0.1-7300-exec-6] Getting password update history with page 1, page size: 2147483647, exclude passwords: true
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.c.a.i.VcfApiTelemetryInterceptor,http-nio-127.0.0.1-7300-exec-3] is public API : true
...
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.s.PasswordUpdateHistoryService,http-nio-127.0.0.1-7300-exec-3] Getting password update history with page 1, page size: 2147483647, exclude passwords: true
yyyy-mm-ddThh:mm:ss DEBUG [vcf_om] [c.v.v.p.helper.NsxtApiUtil,om-exec-26] Failed to update NSX user details : { "error_code" : 403, "module_name" : "common-services", "error_message" : "The credentials were incorrect or the account specified has been locked." } with status : Forbidden
yyyy-mm-ddThh:mm:ss ERROR [vcf_om] [c.v.v.p.helper.NsxtApiUtil,om-exec-26] Recovery of the NSXT service account credential svc-sddc_fqdn-nsxt_fqdn-#### failed with error The credentials were incorrect or the account specified has been locked.

Environment

VCF 9.0

Cause

SDDC has locked the svc account due to multiple incorrect login attempts.

Resolution

Configure the SDDC Manager IP to be immune to lockouts.

• SSH to SDDC manager and run the below command

curl -ks https://nsx_manager_FQDN/api/v1/cluster/api-service -u 'admin:<NSX_admin_password>' | jq ".lockout_immune_addresses += [ \"$(hostname -i)\" ]" | curl -ks https://nsx_manager_FQDN/api/v1/cluster/api-service -u 'admin:<NSX_admin_password>' -X PUT -H 'Content-Type:application/json' -d @-

• Re-try the rotation operation.