NSX-T hostprep fails with the error "Reason: Failed to send HostConfig RPC to MPA"

Products

• VMware NSX

Issue/Introduction

• Hostprep fails with the following error in the NSX-T UI:

"Host configuration: Failed to send the HostConfig message. [TN=TransportNode/933ffb91-####-####-####-305f8ee28405]. Reason: Failed to send HostConfig RPC to MPA TN:933ffb91-####-####-####-305f8ee28405. Error: com.vmware.nsx.management.messaging.exceptions.MessagingException: Unable to reach client 933ffb91-####-####-####-305f8ee28405, application SwitchingVertical. App Init configuration: Failed to send AppInit configuration message. [TN=TransportNode/933ffb91-####-####-####-305f8ee28405]. Reason: Failed to send AppInitConfig RPC to MPA TN:933ffb91-####-####-####-305f8ee28405. Error: com.vmware.nsx.management.messaging.exceptions.MessagingException: Unable to reach client 933ffb91-####-####-####-305f8ee28405, application SwitchingVertical. LogicalSwitch full-sync: LogicalSwitch full-sync realization query skipped."
 

• The nsx-syslog on the esxi reports the following:

2020-07-13T21:09:54Z nsx-proxy: NSX 2104469 - [nsx@6876 comp="nsx-esx" subcomp=nsx-proxy s2comp="nsx-net" tid="2104479" level="WARNING"] StreamConnection[107 Error to ssl://10.##.##.252:1234 sid:-1] Error 34-Numerical result out of range
 

• The /etc/hosts file has the FQDN in Upper case:

/etc/hosts file content:

127.0.0.1       localhost
10.0.0.252      NSXTMGR1.example.com NSXTMGR1

 

• In the nsxapi.log of the manager "NSXTMGR1" you see its own FQDN as upper case and other manager as lower case:

2020-07-14T09:26:12.723Z  INFO ExecutorChannel-1705331988 DnsLookupProviderImpl - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Reverse DNS lookup for IP address 10.##.##.252 found FQDN NSXTMGR1.example.com
2020-07-14T09:26:12.724Z  INFO ExecutorChannel-1705331988 DnsLookupProviderImpl - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Reverse DNS lookup for IP address 10.##.##.253 found FQDN nsxtmgr2.example.com
2020-07-14T09:26:12.719Z  INFO ExecutorChannel-1705331988 DnsLookupProviderImpl - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Reverse DNS lookup for IP address 10.##.##.254 found FQDN nsxtmgr3.example.com

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

During deployment, if the NSX Manager was deployed using upper case FQDN and the DNS entries are in lower case then this results into host preparation issues.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.0.2

Workaround:
There are multiple ways to fix it . 

• Option 1: "Make fqdn false, and restart nsx-proxy service in the host.".

This will make sure that ESXi hosts don't use the NSX FQDN to communicate with the NSX Manager.
On all NSX Managers execute the following:

curl -k -X PUT -H "Content-Type: application/json" -u admin -d '{ "publish_fqdns": false, "_revision": 1}' https://localhost/api/v1/configs/management

On all ESXi hosts execute the following:

/etc/init.d/nsx-proxy restart

• Option 2: Match /etc/hosts file on NSX with the DNS entries

If the /etc/hosts file has the following:

127.0.0.1       localhost
10.##.##.253      NSXTMGR2.example.com NSXTMGR2

Make sure that the DNS entries are also uppercase.