Edge and Transport Nodes Fail to Resolve NSX Manager FQDN ("DNS lookup failed for Manager node" Alarms)

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• Transport Nodes fail to resolve the NSX Manager Fully Qualified Domain Name (FQDN) and generate DNS lookup failures in their logs.
  
  
• The following alarms are triggered in the NSX Manager UI:
  
  
• DNS lookup failed for Manager node <uuid> with FQDN <fqdn>.
  
  
• Reverse DNS lookup failed for Manager node <uuid> with IP address <ip-address>.
  
  
• Command Discrepancy: Running nslookup <fqdn> generates the correct output, but dig fails. The outputs of dig <fqdn> and dig -x <ip> do not contain an "Answer" section.

Environment

• VMware NSX-T Data Center (Version 3.2.3 and higher)
• VMware NSX

Cause

DNS resolution failures in this context typically occur due to one of the following reasons:

1. DNS Configuration Issues: Incorrect or missing A/PTR DNS records for the NSX Manager or vCenter, or the DNS server is misconfigured/unavailable.
   
   
2. Hostname Resolution Failures: NSX Manager is unable to resolve the vCenter server's hostname to an IP address.
   
   
3. Firewall or Security Group Blocking: Network traffic (UDP/TCP Port 53) between the NSX Manager/Nodes and the DNS server is blocked.
   
   
4. Certificate Trust Validation: Issues with SSL/TLS certificates can cause connection failures if the connection relies on strict FQDN resolution for trust validation.

Resolution

To verify and resolve the DNS resolution failure, perform the following troubleshooting steps on the affected node:

Step 1: Verify Forward Lookup using dig Run the following command to test forward DNS resolution.

Verify that the output contains the correct IP in the ANSWER SECTION.

• • Example of a correct output:

    • root@edge01:~# /usr/bin/dig nsx-mngr-01.#.#
      ; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> nsx-mngr-01.#.#
      ;; global options: +cmd
      ;; Got answer:
      ;; WARNING: .local is reserved for Multicast DNS
      ;; You are currently testing what happens when an mDNS query is leaked to DNS
      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15328
      ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
      
      ;; OPT PSEUDOSECTION:
      ; EDNS: version: 0, flags:; udp: 4000
      ;; QUESTION SECTION:
      ;nsx-mngr-01.#.#.                IN      A
      
      ;; ANSWER SECTION:
      nsx-mngr-01.#.#. 3600    IN      A       192.#.#.#
      
      ;; Query time: 0 msec
      ;; SERVER: #.#.#.10#53(192.#.#.#) (UDP)
      ;; WHEN: Thu Feb 20 14:44:32 UTC 2025
      ;; MSG SIZE  rcvd: 67

Step 2: Verify Reverse Lookup using dig Run the following command to test reverse DNS resolution.

Verify that the output contains the correct FQDN in the ANSWER SECTION.

Run the /usr/bin/dig -x <IP-Address> command, and verify the output in the "Answer" section is correct:

• • Example of a correct output:

    • root@edge01:~# /usr/bin/dig -x 192.#.#.#
      ; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> -x 192.#.#.#
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30823
      ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
      
      ;; OPT PSEUDOSECTION:
      ; EDNS: version: 0, flags:; udp: 4000
      ;; QUESTION SECTION:
      ;10.#.#.#.in-addr.arpa.   IN      PTR
      
      ;; ANSWER SECTION:
      10.#.#.#.#. 3600 IN    PTR     controlcenter.#.#.
      
      ;; Query time: 4 msec
      ;; SERVER: #.#.#.10#53(#.#.#.#) (UDP)
      ;; WHEN: Thu Feb 20 15:02:32 UTC 2025
      ;; MSG SIZE  rcvd: 94

Step 3: Analyze and Remediate Connectivity

• If the outputs in Steps 1 and 2 are missing the ANSWER SECTION or show incorrect values, the issue lies with the DNS server configuration or network connectivity.
• Perform a packet trace for queries sent to and from the DNS server (port 53) to confirm the DNS server is successfully receiving and replying to the queries without packet drops.

Additional Information

• Periodic Checks: When the publish_fqdns flag is set to True, the system periodically runs the /usr/bin/dig command for FQDN resolution. Failures in this background check generate log errors, triggering the DNS failure alarm in the NSX Manager UI.
  
  
• Resolution Sequence: NSX nodes attempt to resolve FQDNs in the following strict fallback sequence:
1. /usr/bin/dig
2. nslookup
3. /usr/bin/getent hosts
   
   
• Changes in 3.2.3+: While ESXi hosts rely on nslookup, the dig command was introduced as the primary tool in NSX-T Data Center 3.2.3. Depending on the specific build, edge nodes may fall back to getent or nslookup if dig is absent.

Note: The local /etc/hosts file is not used for DNS resolution in NSX-T Data Center 3.2.3 or higher.