"One or more DNS server is not reachable or File server IP and FQDN not matching with DNS entries." warning on vSAN File Services

Products

• VMware vSAN

Issue/Introduction

Symptoms:

vSAN Skyline Health shows a failed test under 'File Service → File Server Health → File Server Connectivity' with the "One or more DNS server is not reachable or File server IP and FQDN not matching with DNS entries." error.

Based on this screenshot we can see vSAN is having issues when doing 'DNS lookup' test commands against the container with IP address aa.bb.cc.dd.

Ping the DNS server from the container works.

Resolving forward and reverse 'nslookup' commands from the DNS server to the container. 

The following or similar entries on the host where the FSVM with the container in question is running:

→ /var/run/log/vsanfs.mgmt.log

2022-04-27T20:27:22.963Z info [EndpointMonitor-2] [_CheckContainerHealthForMgmt] CONT: start checking on aa.bb.cc.dd
2022-04-27T20:27:22.963Z info [EndpointMonitor-2] [_ExecRun] Running cmd nslookup -timeout=5 vsanfs04.domain_fqdn aa.bb.cc.ee on container None(vsanfs04), with timeout: 6
2022-04-27T20:27:24.10Z info [EndpointMonitor-2] [_ExecRun] Output: Server: aa.bb.cc.ee Address: aa.bb.cc.ee Name: vsanfs04.domain_fqdn Address: aa.bb.cc.dd
2022-04-27T20:27:24.11Z info [EndpointMonitor-2] [_ExecRun] Running cmd nslookup -timeout=5 aa.bb.cc.dd aa.bb.cc.ee on container None(vsanfs04), with timeout: 6
2022-04-27T20:27:25.66Z info [EndpointMonitor-2] [_ExecRun] Output: dd.cc.bb.aa.in-addr.arpa name = VSANFS04.domain_fqdn.
2022-04-27T20:27:25.67Z warning [EndpointMonitor-2] [_CheckContainerHealthItem] CONT: Error running DnsLookupTest on container aa.bb.cc.dd: False, Reverse DNS lookup not working or incorrect.

Environment

vSphere ESXI 8.x

Cause

• The issue is caused by a DNS record entries mismatch in the FSVM FQDN between what the user configured on the DNS server and vCenter.

• The vSANFS endpoint controller will log a "Error running DnsLookupTest on container X.X.X.X.: False, Reverse DNS lookup not working or incorrect." when doing a reverse lookup zone test because it is case sensitive. 

For example:

FSVM FQDN configured on vCenter: vsanfs04.domain_fqdn

FSVM FQDN configured on the forward lookup zone in DNS server: vsanfs04.domain_fqdn

FSVM FQDN configured on the reverse lookup zone in DNS server: VSANFS04.domain_fqdn

Forward lookup resolves fine since there is no CAPS mismatch.

2022-04-27T20:27:22.963Z info [EndpointMonitor-2] [_ExecRun] Running cmd nslookup -timeout=5 vsanfs04.domain_fqdn aa.bb.cc.ee on container None(vsanfs04), with timeout: 6
2022-04-27T20:27:24.10Z info [EndpointMonitor-2] [_ExecRun] Output: Server: aa.bb.cc.ee Address: aa.bb.cc.ee Name: vsanfs04.domain_fqdn Address: aa.bb.cc.dd

Reverse lookup fails getting resolved because the FQDN in DNS server has the record in uppercase:

2022-04-27T20:27:24.11Z info [EndpointMonitor-2] [_ExecRun] Running cmd nslookup -timeout=5 aa.bb.cc.dd aa.bb.cc.ee on container None(vsanfs04), with timeout: 6
2022-04-27T20:27:25.66Z info [EndpointMonitor-2] [_ExecRun] Output: dd.cc.bb.aa.in-addr.arpa name = VSANFS04.domain_fqdn.  <<<<<========== Refer this.

2022-04-27T20:27:25.67Z warning [EndpointMonitor-2] [_CheckContainerHealthItem] CONT: Error running DnsLookupTest on container aa.bb.cc.dd: False, Reverse DNS lookup not working or incorrect.

• When reviewing the Reverse Lookup zone settings on the DNS server side you should see the FQDN with the case mismatch. In this case, note the difference between the FSVM failing and the rest:

Resolution

This issue has been fixed on vSphere 7.0 U3d release.

The steps to permanently fix this issue are the following:

1. Upgrade both vCenter and ESXi to 7.0 U3d.
2. After finishing with the vCenter and ESXi updades, update the vSAN On-Disk format version to the latest.
3. Upgrade the FSVM version to the latest possible one. Refer to the following link for more details: Upgrade File Service 

Workaround:

If an upgrade is not possible at the moment, the customer could modify the DNS FQDN entries to match the ones configured on vCenter/vSAN.