HCX VM migration fails with error: "Could not resolve segment" since it is "Unable to contact the authentication service."

Products

• VMware NSX
• VMware HCX

Issue/Introduction

• When a VM migration fails a report similar to the following is reported in the HCX UI:

vMotion failed. Could not resolve segment /infra/segments/hcx-ne-<UUID> to opaque network. Failed to get realized state. Result: {"status":"failure","statusCode":504,"details":"","result":{"module_name":"common-services","error_message":"Unable to contact the authentication service.","sub_error":"2","error_code":408}}

• A log similar to the following is reported in the HCX /common/logs/admin/app.log:

UTC [VmotionService_SvcThread-61933, Ent: HybridityAdmin, , TxId: TxId: <UUID>] ERROR c.v.h.s.v.j.StartTargetSideRelocateVmWorkflow- [migId=<UUID>
94] Error while executing startTargetSideRelocateVmWorkflow state 'START_RELOCATE_VM'.
java.lang.RuntimeException: Could not resolve segment /infra/segments/hcx-ne-<UUID> to opaque network. Failed to get realized state. Result: {"status":"failure","statusCode":504,"details":"","result":{"module_name":"common-services","error_message":"Unable to contact the authentication service.","sub_error":"2","error_code":408}}

• At the same timestamp in the NSX Manager /var/log/proxy/reverse-proxy.log something similar to the following will be reported, which indicates that there is a timeout when trying to connect to the LDAP server:

INFO grpc-default-executor-195110 HttpClientUtil 84227 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Making request to http://127.0.0.1:6565/policy/api/v1/infra/realized-state/realized-entities?intent_path=%2Finfra%2Fsegments%2Fhcx-ne-<UUID>
WARN Processing request <UUID> DelegatingLdapAuthProvider 84227 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] Cannot connect to LDAP server: Read timed out 
org.springframework.ldap.CommunicationException: <LDAP Server FQDN>:636; nested exception is javax.naming.CommunicationException: <LDAP Server FQDN>:636 [Root exception is java.net.SocketTimeoutException: Read timed out]
        at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:108) ~[spring-ldap-core-2.4.1.jar:2.4.1]

• In the NSX Manager /var/log/syslog it will be reported that the login for the user is a failure:

<NSX Manager FQDN> NSX 84227 SYSTEM [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="http"] UserName="<USERNAME USED BY HCX TO LOGIN TO NSX MANAGER>@<IP ADDRESS>", ModuleName="ACCESS_CONTROL", Operation="LOGIN", Operation status="failure"

Environment

• VMware NSX
• VMware HCX

Cause

• Authentication or communication issue between the NSX Manager and the LDAPS servers.

Resolution

• As a workaround reconfigure HCX use the local NSX admin account instead of a domain account until the issue with the LDAP server is resolved:
  1. Ensure you have the correct NSX admin password and are able to login to all of the NSX managers in the NSX cluster with this account
  2. Login to HCX VAMI page in the Target Cloud environment (https://(hcx-manager.fqdn):9443) using the HCX admin user account
  3. On the HCX Dashboard, Click "Manage" for the NSX Tile, then Click "Edit"
  4. Update the NSX admin account credentials and click "Save"
  5. Go to the "Appliance Summary" tab and restart "Web Service" and "App Service" Services (or reboot the HCX appliance). Click "Stop", then click 'Start" when available.
• If the requirement is to login from HCX using an LDAP account then the following steps may help with identifying the cause:
  • Latency Check: Verify the latency between the NSX Manager and LDAP Server by pinging from one to the other.
  • DNS: Confirm from the NSX Manager that it can resolve the FQDN of the LDAP Server.
  • LDAP Server Health: Check the LDAP server logs at the timestamp of the last failed login. See if the LDAP server received the request or was under heavy load.