False DNS Forwarder Upstream Server Timeouts in NSX-T 3.2.2 or later

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You have upgraded to NSX-T version 3.2.2 or later.
• In the NSX UI you receive alarms for "Forwarder Upstream Server Timeout".
• DNS is not affected and is working as expected.
• In var/log/dns/failed-query.log on the Edge node logs we see the following error:

2022-11-22T09:47:18.944Z Edge01.corp.local dnsmasq_########-####-####-####-########d9b8 25 - -  Failed DNS Query. record_type: 1, address: ns-02.corp.local, error_type: TIMEOUT, client_ip: 192.168.10.10, forwarder_ip: 192.168.200.10, source_ip: 192.168.100.10, upstream_server_ip: 8.8.8.8, time_spent: 65, error_message: None,

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

 

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

This is due to a change in the event/alarm algorithm introduced in NSX-T 3.2.2, which results in false positive DNS timeout alarms.

Resolution

This is a known issue currently impacting NSX-T.

Workaround:

The DNS forwarder timeout alarm in the NSX Alarm tab can be manually disabled. To achieve this do the following from the NSX UI: 

1. Browse to: Alarms > Alarm Definitions
2. Highlight the alarm in question
3. Click on the vertical ellipsis and click "Edit"
4. Deselect the checkbox for "Enable Detection"

Note: If there is a real condition causing DNS issues, you will not be alerted.

When you have upgraded to a version where this issue has been resolved, revert the above changes.