vCenter WCP service unable to start due to NSX-T FQDN invalid character invalid character in host name or unable to reach DNS server configured on NSX manager(s)

Products

• VMware vCenter Server
• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

vCenter is registered as a compute manager in NSX if an invalid FQDN returns for the NSX manager VIP WCP will crash.  WCP will also not start if NSX manager(s) can't reach a configured DNS server.

Cause

If PTR reverse records for the NSX IP have a space at the end or another invalid character WCP will not start.  In the below example the DNS server had a space at the end.

vCenter logging

var/log/vmware/wcp/wcpsvc.log

38715:2024-09-27T17:34:51.46Z error wcp [nsxtlib/utils.go:204] Failed to parse NSX extension server URL; https://<NSX VIP FQDN>\032:443, err: parse "https://<NSX VIP FQDN>\\032:443": invalid character "\\" in host name.

var/log/vmware/wcp/stdstream.log.stderr

level=error msg="Received error seeding nsxManagersCache: failed to parse NSX server url: parse \"https://<NSX FQDN>\\\\032:443\": invalid character \"\\\\\" in host name"
level=fatal msg="Unable to initialize NSX extension monitor: failed to parse NSX server url: parse \"https://<NSX FQDN>\\\\032:443\": invalid character \"\\\\\" in host name"

NSX manager logging

var/log/cm-inventory/cm-inventory.log

INFO ClusterNodeConfigListener-1-1 Utils 4809 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] getFqdnFromIp(): script returned FQDN: <NSX VIP FQDN>\032
INFO ClusterNodeConfigListener-1-1 IpAddressUtils 4809 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] getFqdnFromIp(): returning FQDN : <NSX VIP FQDN>\032 for IP: <x.x.x.x>

In the above logging examples \032 is at the end of the FQDN reverse resolution because the space is \032 in ascii.

dig -x from linux will show the 032

nslookup in windows will NOT show the 032, just a space that won't be visible in the cmd prompt

You may also experience intermittent or a slow response and/or no response from any of their DNS server(s). If they have multiple DNS records (potentially in a Round-Robin configuration), be sure to check both or all DNS servers that are listed within the /etc/resolv.conf files.

i.e. nslookup <host fqdn> <dns ipaddress> 

You may observe a DNS SERVFail 'timeout' error such as 'connection timed out; no servers could be reached' or 'not available'.

If NSX manager can't reach a configured DNS server the vCenter's var/log/vmware/wcp/wcpsvc.log may exhibit the following error:

error wcp [nsxtlib/utils.go:204] Failed to parse NSX extension server URL; https://;; communications error to <Unreachable DNS IP>#53: timed out:443 err: parse "https://;; communications error to <Unreachable DNS IP>": invalid character " " in host name

And you will observe a failure when attempting to start the vCenter's WCP service (which is why the vCenter upgrade is failing as observed introduction above):

vmon-cli --status wcp
vmon-cli --start wcp

Resolution

Ensure forward and reverse entries contain only valid DNS characters and formatting. 

Validate that NSX DNS is pointing to the appropriate servers. This is a returned error in WCP from NSX and vCenter. 

Configure NSX Manager for Access by DNS Server

You may also need to remove and re-add the PTR records for all three NSX Manager Nodes and the VIP within the DNS or completely remove the bad/failed DNS server entry from all of the affected components.