HCX Service Mesh Alterations or Migrations Fail Due to NSX-T API Authentication Failure

Products

• VMware HCX
• VMware NSX Networking

Issue/Introduction

Symptoms:

• HCX Is integrated with NSX-T on either the Connector or Cloud side.
• The password of the NSX-T admin account used by HCX for API authentication was recently changed.
• Service Mesh Operations (Deploy, Resync, Edit) fail with error stating 'Could not fetch NSX-T Transport Zone from the Compute configured in the Compute Profile'. 
• HCX Migration operations fail with error stating 'Could not resolve segment /infra/segments/<network-name>/'.
• Errors in the app.log (/common/logs/admin/app.log) of the HCX Manager integrated with NSX-T resemble:

2021-08-03 16:06:51.431 UTC [NSXService_SvcThread-23360, Ent: HybridityAdmin, , TxId: 4xxxxxx4-7xx9-4xxb-8xx5-9xxxxxxxxxx7] ERROR c.v.v.h.s.n.NsxTInventorySyncJob- Error while syncing the traffic groups. Got
Response:{"status":"failure","statusCode":403,"details":"","result":{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403
}}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

By default, after five consecutive failed attempts authenticate to NSX-T, the account is locked for 15 minutes. This lock is enforced by source IP, thus only source IPs trying to authenticate via incorrect credentials will be locked. If the password is changed for the admin account of NSX-T and not immediately updated within HCX, HCX will lock itself out within minutes due to the frequency of API calls that HCX sends to NSX-T.

Resolution

To remedy this, you may either:

1.) Change the NSX-T API Authentication Policy. 

Steps to do so are: 

• Login to the NSX-T Managers via SSH utilizing the 'admin' account to reach the central CLI.
• Run the command 'set auth-policy api lockout-period 0'.
• Navigate to the HCX Administrator page  (https://<HCX-FQDN-OR-IP>:9443) and enter the updated NSX-T admin credentials.
• Verify that HCX can now perform Service Mesh Operations or Migrations.
• (Optional) Set the NSX-T Authentication Policy back to default via set auth-policy api lockout-period 15

More information on the NSX-T Authentication Policy options may be found here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.0/administration/GUID-99BAED85-D754-4589-9050-72A1AB528C10.html
 

2.) Update the password within HCX and let the Authentication Lockout Period elapse.

Steps to do so are: 

• Navigate to the HCX Administrator page  (https://<HCX-FQDN-OR-IP>:9443) and enter the updated NSX-T admin credentials.
• Power down the HCX Manager VM for 15 minutes (or to match the duration of the NSX-T Authentication Policy, if it's been altered from its default of 15 minutes).
  • This is to prevent any API calls from the HCX Manager to NSX-T while the lockout elapses. 
• Once the lockout period has elapsed, power the HCX Manager VM back on
• Verify that HCX can now perform Service Mesh Operations or Migrations