HCX Service Mesh Alterations, Migrations, or update of integrated NSX account credentials fail due to NSX-T API Authentication failure

Products

VMware HCX VMware NSX

Issue/Introduction

HCX is integrated with NSX-T on either the Connector or Cloud side.
The password of the NSX-T admin account used by HCX for API authentication was recently changed.
Service Mesh Operations (Deploy, Resync, Edit) fail with error stating "Could not fetch NSX-T Transport Zone from the Compute configured in the Compute Profile".
HCX Migration operations fail with error stating "Could not resolve segment /infra/segments/<network-name>/".
Update of the NSX admin account credentials from within the HCX Administrator Web portal (https://<HCX-FQDN-OR-IP>:9443) results in a failure message: "The credentials are incorrect or the account specified has been locked".
Errors in the app.log (/common/logs/admin/app.log) of the HCX Manager integrated with NSX display the error:

2021-08-03 16:06:51.431 UTC [NSXService_SvcThread-23360, Ent: HybridityAdmin, , TxId: ########-####-####-####-############] ERROR c.v.v.h.s.n.NsxTInventorySyncJob- Error while syncing the traffic groups. Got
Response:{"status":"failure","statusCode":403,"details":"","result":{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403
}}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware HCX
VMware NSX

Cause

By default, after five consecutive failed attempts to authenticate to NSX-T, the NSX admin account is locked for 15 minutes. This lock is enforced by source IP, thus only source IPs trying to authenticate via incorrect credentials will be locked. If the password is changed for the admin account of NSX-T and not immediately updated within HCX, HCX will lock itself out within minutes due to the frequency of API calls that HCX sends to NSX-T.

Resolution

To remedy this, you may either:

A.) Change the NSX-T API Authentication Policy.

Steps to do so are:

Login to the NSX-T Managers via SSH, utilizing the 'admin' account to reach the central CLI.
Run the command: 'set auth-policy api lockout-period 0'.
Navigate to the HCX Administrator portal: (https://<HCX-FQDN-OR-IP>:9443) and enter the updated NSX-T admin credentials.
Verify that the updated admin account credentials were accepted by navigating to the "Dashboard" page within the same Administrator portal and confirming that NSX shows as being healthy, with a small green circle visible within the NSX section:
Verify that HCX can now perform Service Mesh Operations or Migrations.
Set the NSX-T Authentication Policy back to default by using the CLI command: 'set auth-policy api lockout-period 15'.

B.) Change the NSX API lockout reset period.

Login to the NSX-T Managers via SSH, utilizing the 'admin' account to reach the central CLI.
Use get auth-policy api lockout-reset-period to check the current configuration. The default is 900.
Use set auth-policy api lockout-reset-period 5 to reduce the lockout reset period.
Try entering the NSX credentials into the HCX webadmin (https://<HCX Manager FQDN>:9443) again.
Reset back to the original lockout-period configuration with set auth-policy api lockout-reset-period 900.
Please see Admin account lockout policy for the NSX Edge and Manager nodes for more information.

C.) Update the password within HCX and let the Authentication Lockout Period elapse.

Steps to do so are:

Navigate to the HCX Administrator portal (https://<HCX-FQDN-OR-IP>:9443) and enter the updated NSX-T admin credentials. You may see a failure message stating: "The credentials are incorrect or the account specified has been locked". This can be ignored, as like the message says, the admin account is currently locked.
Power down the HCX Manager VM for 15 minutes (or to match the duration of the NSX-T Authentication Policy if it's been altered from its default of 15 minutes). .
- This is to prevent any API calls from the HCX Manager to NSX-T from making authentication attempts while the lockout elapses and directly after the lockout has elapsed
Once the lockout period has elapsed, power the HCX Manager VM back up.
Verify that the updated admin account credentials were accepted by navigating to the "Dashboard" page within the HCX Administrator portal and confirming that NSX shows as being healthy, with a small green circle visible within the NSX section. (See image above)
Verify that HCX can now perform Service Mesh Operations or Migrations.

Additional Information

More information on the NSX Authentication Policy options can be found here: Authentication Policy Settings.