Unable to deploy VCF Workload Domain with NFS Datastore - Failed to validate NFS configuration on port: 111 Connection timed out due to wrong vmnic mapping

Products

• VMware SDDC Manager

Issue/Introduction

• The Workload Domain workflow fails with on the NFS Datastore Configuration validation with the following error message:
  
  Failed to validate NFS configuration for xxx-xxx-xxx-xxx : Operation failed, diagnostics report: Error: 0x202: Unable to reach xxx-xxx-xxx-xxx on port: 111, Connection timed out


• Logs showing the following error:
  
  /var/log/vmware/vcf/domainmanager/domainmanager.log
  yyyy-mm-ddThh:mm:ss.zzz+0000 ERROR [vcf_dm,<task_id>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10]  [######] VSPHERE_NFS_VALIDATION_FAILED Failed to validate NFS datastore at
  <NFS_IP> with path /mnt on host <ESXI> : Operation failed, diagnostics report: Error: 0x202: Unable to reach <NFS_IP> on port: 111,  Connection timed out
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate NFS datastore at <NFS_IP> with path /mnt on host <ESXI> : Operation failed, diagnostics report: 
  Error: 0x202: Unable to reach <NFS_IP> on port: 111,  Connection timed out
  
  
• The network configuration has already been validated using KB418803
  
  
• During the deployment we observe an incorrect mapping of the uplinks with vmnics in the NFS vmkernel configuration
  
  
  Example given: 
  On the following configuration we expect based on the uplink mapping for the NFS portgroup to be assigned to vmnic0 (uplink1). 
  Instead during the Workload Domain deployment the  NFS portgroup is wrongly assigned to vmnic3.
  
  DVS configuration:
=======================
MTU: 9000
Number of uplinks: 4
uplink1: vmnic0
uplink2: vmnic1
uplink3: vmnic2
uplink4: vmnic3
=======================d
NFS configuration:
Portgroup: NFS
Active Uplinks: uplink1
Standby Uplinks: uplink2
Unused Uplinks: uplink3, uplink4
  
  Actual NFS Deployment configuration: vmnic3  # ===> Wrong mapping results in incorrect network configuration

Environment

VMware Cloud Foundation 9.0.0
VMware Cloud Foundation 9.0.1

Cause

This is a known issue of the SDDC Manager affecting the workload domain with NFS datastore workflow. 

Resolution

Engineering is aware of this issue and working to resolve this in a future release.


Workaround:

1. Take a snapshot of the SDDC Manager VM.
   
   
2. SSH into the SDDC VM and elevate to root.
   
   
3. Take a backup of the application.properties file:
   
   cp /etc/vmware/vcf/domainmanager/application.properties /etc/vmware/vcf/domainmanager/application.properties.backup

4. Open the file /etc/vmware/vcf/domainmanager/application.properties with a text editor and add the following line:
   
   fsm.ValidateHostNfsDataStoreAction.skipCheck=true
   
   
5. Save the file.
   
   
6. Restart the domainmanager service:
   
   systemctl restart domainmanager
   
   
7. Reattempt the workflow.

Additional Information

"Failed to validate NFS datastore" error during VCF 9 new workload domain deployment

VCF 9 Deployment Validation Fails on NFS Storage – NFS Datastore Not Found on Host