"Failed to validate NFS datastore" error during VCF 9 new workload domain deployment

Products

• VMware vSphere ESXi

Issue/Introduction

• When configuring the new workload domain, the selected hosts have 4 vmnic's, vmnic0 is mapped to the standard vSwitch for management.

• During the deployment, at the Distributed Switch step, the 4 vmnics are selected and mapped:

uplink1 vmnic0
uplink1 vmnic1
uplink1 vmnic2
uplink1 vmnic3

• When creating the vmkernel for the NFS datastore, uplink1 is selected as Active and uplink 2 is selected as Standby.

• The deployment fails with error:

  Failed to validate NFS datastore at <nfs-data-store-ip-address> with path <mount-point> on <esxi-host>: Operation failed, diagnostics report:
  Error: 0x202: Unable to reach <nfs-data-store-ip-address> on port 111, Connection timed out.

• This error can also be seen in the SDDC manager log /var/log/vmware/vcf/domainmanager/domainmanager.log:

message":"Failed to validate NFS datastore at <nfs-datastore-ip-address> with path <nfs-mount-point> on host <esxi-host> : Operation failed, diagnostics report: Error: 0x202: Unable to reach <nfs-datastore-ip-address> on port: 111, Connection timed out","remediationMessage":"Ensure that provided NFS details are correct","referenceToken":"######"}]

Environment

VCF 9

Cause

The vmnic(s) used in the uplink, for the NFS vmkernel, may not have the correct VLAN present on the trunk port of the physical switch.

Resolution

Confirm with the physical networking team, that the trunk port on the physical switch which connects to the ESXi host vmnics (vmnic0 and vmnic1 in this case), permit the NFS VLAN.

To test the host has the correct VLAN for the vmnic, create a test vmkernel on the standard switch, using the VLAN required and add to the current standard switch or create new standard switch.

• Log into the ESXi host web interface, go to Networking, select Vmkernel NICs and click Add.
• Populate the details, making sure to use the VLAN and an IP address which is able to connect to the NFS datastore, the same details used in the SDDC new workload domain creation workflow, and save.
• Then log in to the ESXi host as root user and check the new vmkernel is created:

esxcfg-vmknic -l

• Next, use netcat to test connectivity to the destination NFS datastore, using the new NFS vmkernel:

nc -zv <nfs-ip-address> 111 -s <nfs-vmkernel-ip-address>

• If port 111 is open to the NFS datastore, you should expect to get a 'succeeded'.

Note: Port 111 is a portmapper port, used by SDDC manager to validate NFS datastore connectivity, please review the required ports:

VMware Ports and Protocols

For additional details on how to check allowed VLAN's on ESXi hosts, please review the following KB VLAN configuration on virtual switches, physical switches, and virtual machines