NSX-V 6.4.8 and 6.4.9 Edge gateways configured for DNS forwarding may return SERVER FAILURE (SERVFAIL) status to clients despite a successful DNS response from the DNS server

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• VMs using an NSX Edge for DNS does not work after a new deployment of NSX-V 6.4.8
• VMs using an NSX Edge for DNS does not work after upgrading the Edge to NSX-V 6.4.8
• A packet capture taken on the NSX Edge will show the Edge reports SERVER FAILURE status to DNS clients despite the Edge receiving a successful DNS response from the DNS server
• The logs on the Edge gateway will show messages similar to the following:

2020-10-16T16:38:45+00:00 nsxv-edge-0 dns[2053]: [default]:  [url.info] info: no valid RRSIG resolving 'org/DS/IN': <DNS IP>#53
2020-10-16T16:38:45+00:00 nsxv-edge-0 named[2053]: [default]:  [url.info] info: no valid RRSIG resolving 'org/DS/IN': <DNS IP>#53
...
2020-10-16T16:38:45+00:00 nsxv-edge-0 dns[2053]: [default]:  [url.debug] debug 1: client @0x6c1604042580 <client IP>#41362 (<DNS hostname>): view vsm-default-view: query failed (SERVFAIL) for <DNS hostname>/IN/A at query.c:6801
2020-10-16T16:38:45+00:00 nsxv-edge-0 named[2053]: [default]:  [url.debug] debug 1: client @0x6c1604042580 <client IP>#41362 (<DNS hostname>): view vsm-default-view: query failed (SERVFAIL) for <DNS hostname>/IN/A at query.c:6801

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

The BIND package used for DNS forwarding functionality in NSX-V 6.4.8 and 6.4.9 has DNSSEC validation enabled by default.  The SERVFAIL status may be returned by the Edge if the DNS server does not support DNSSEC leading to DNS resolution failures for clients.

Resolution

This issue has been resolved in 6.4.11

Workaround:
To workaround the issue, disable DNSSEC validation on the NSX Edge:

1. Connect to the NSX Manager as admin and enter enable mode by typing: en

2. Enter engineering mode by typing: st en

3. Enter the NSX Manager root password: IAmOnThePhoneWithTechSupport
4. Get the password for the Edge by typing: /home/secureall/secureall/sem/WEB-INF/classes/GetSpockEdgePassword.sh
5. Access the Edge VM console, log in as the admin user and enter enable mode by typing: en
6. Enable engineering mode by typing: debug engineeringmode enable
7. Enter the root shell on the Edge by typing and using the password from step #4: st en
8. Modify the file /var/db/vmware/vshield/dns.conf and add the following lines under the options section:

  dnssec-enable no;
  dnssec-validation no;

9. Restart DNS service:

/etc/init.d/bind9 restart