vCenter WCP service unable to start due to NSX-T FQDN invalid character invalid character in host name or unable to reach DNS server configured on NSX manager(s)

Products

• VMware vCenter Server
• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

vCenter is registered as a compute manager in NSX if an invalid FQDN returns for the NSX manager VIP WCP will crash.  WCP will also not start if NSX manager(s) can't reach a configured DNS server.

Cause

If PTR reverse records for the NSX IP have a space at the end or another invalid character WCP will not start.  In the below example the DNS server had a space at the end.

vCenter logging

var/log/vmware/wcp/wcpsvc.log

38715:2024-09-27T17:34:51.46Z error wcp [nsxtlib/utils.go:204] Failed to parse NSX extension server URL; https://<NSX VIP FQDN>\032:443, err: parse "https://<NSX VIP FQDN>\\032:443": invalid character "\\" in host name.

var/log/vmware/wcp/stdstream.log.stderr

level=error msg="Received error seeding nsxManagersCache: failed to parse NSX server url: parse \"https://<NSX FQDN>\\\\032:443\": invalid character \"\\\\\" in host name"
level=fatal msg="Unable to initialize NSX extension monitor: failed to parse NSX server url: parse \"https://<NSX FQDN>\\\\032:443\": invalid character \"\\\\\" in host name"

NSX manager logging

var/log/cm-inventory/cm-inventory.log

INFO ClusterNodeConfigListener-1-1 Utils 4809 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] getFqdnFromIp(): script returned FQDN: <NSX VIP FQDN>\032
INFO ClusterNodeConfigListener-1-1 IpAddressUtils 4809 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] getFqdnFromIp(): returning FQDN : <NSX VIP FQDN>\032 for IP: <x.x.x.x>

In the above logging examples \032 is at the end of the FQDN reverse resolution because the space is \032 in ascii.

dig -x from linux will show the 032

nslookup in windows will NOT show the 032, just a space that won't be visible in the cmd prompt

You may also experience intermittent or a slow response and/or no response from any of their DNS server(s). If they have multiple DNS records (potentially in a Round-Robin configuration), be sure to check both or all DNS servers that are listed within the /etc/resolv.conf files.

i.e. nslookup <host fqdn> <dns ipaddress> 

You may observe a DNS SERVFail 'timeout' error such as 'connection timed out; no servers could be reached' or 'not available'.

If NSX manager can't reach a configured DNS server the vCenter's var/log/vmware/wcp/wcpsvc.log may exhibit the following error:

error wcp [nsxtlib/utils.go:204] Failed to parse NSX extension server URL; https://;; communications error to <Unreachable DNS IP>#53: timed out:443 err: parse "https://;; communications error to <Unreachable DNS IP>": invalid character " " in host name

And you will observe a failure when attempting to start the vCenter's WCP service (which is why the vCenter upgrade is failing as observed introduction above):

vmon-cli --status wcp
vmon-cli --start wcp

Resolution

Ensure forward and reverse entries contain only valid DNS characters and formatting. 

Configure NSX Manager for Access by DNS Server

You may also need to remove and re-add the PTR records for all three NSX Manager Nodes and the VIP within the DNS or completely remove the bad/failed DNS server entry from all of the affected components.