VCF 9.X Management Domain deployment on NFS with separate DVS for storage fails with VSPHERE_CREATE_NFS_VSWITCH

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting a deployment of a VMware Cloud Foundation (VCF) Management Domain on NFS with separate DVS configuration for storage in deployment spec, the deployment fails during the Create NFS vSwitch stage with the below error:

VSPHERE_CREATE_NFS_VSWITCH_FAILED Failed to create NFS vSwitch on host <esxi_fqdn_with_NFS_mounted>
The domainmanager.log on the VCF installer shows entries like below:

/var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,<taskID>,<subtaskID>] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-192] Assigning default gateway ####.####.####.#### for portGroup <NFS_portGroup>
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<taskID>,<subtaskID>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-192] [ReferenceToken#] VSPHERE_CREATE_NFS_VMNICS_FAILED Failed to create NFS VMKernel adapter under DVS <esxi_fqdn_with_NFS_mounted>
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to create NFS VMKernel adapter under DVS <esx_fqdn_with_NFS_mounted>
...
Caused by: com.vmware.vim.binding.vim.fault.PlatformConfigFault: An error occurred during host configuration
...
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<taskID>,<subtaskID>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-587] [ReferenceToken#] VSPHERE_CREATE_NFS_VSWITCH_FAILED Failed to create NFS vSwitch <Storage_NFS_VDS> on host <esxi_fqdn_with_NFS_mounted>
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to create NFS vSwitch <Storage_NFS_VDS> on host <esxi_fqdn_with_NFS_mounted>
        at com.vmware.vcf.common.fsm.plugins.action.impl.CreateNfsVmknicOnHost.execute(CreateNfsVmknicOnHost.java:145)
        at com.vmware.vcf.common.fsm.plugins.action.impl.CreateNfsVmknicOnHost.execute(CreateNfsVmknicOnHost.java:68)...Caused by: com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereOperationException: Unable to add vSwitch <Storage_NFS_VDS> on <esxi_fqdn_with_NFS_mounted>
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.addVswitch(VcManagerBase.java:5971)
        at com.vmware.vcf.common.fsm.plugins.action.impl.CreateNfsVmknicOnHost.execute(CreateNfsVmknicOnHost.java:138)

Environment

VCF 9.x

Cause

When the VCF installer is configured to become the SDDC Manager by setting the useExistingDeployment flag to true in the Management domain deployment specification, the deployment fails if the NFS VMkernel port group is preconfigured on the management virtual switch. The VCF installer strictly expects this port group to be configured on a separate, dedicated vSwitch on the ESX host where it resides, with a port group name matching the input network specification and a vSwitch name matching the NFS DVS name specification. Furthermore, post failed deployment attempt, VCF installer does not clean up stale traces, causing subsequent attempts to fail.

Resolution

When useExistingDeployment is set to true and the NFS vmknic is being preconfigured on the first ESX host where the VCF Installer resides, ensure the configuration strictly matches the input specification by following the below steps:

NOTE: In case the VCF Installer resides on the NFS share that is mounted on the ESXi host, proceed with a fresh deployment attempt instead of a retry, as the NFS share needs to be unmounted and remounted properly.

Review and determine the stale vswitch and vmkernel adapters present:

esxcfg-vswitch -l
esxcli network vswitch standard list
esxcli network ip interface list
Delete any stale VMkernel port group(s) and vswitch(s) from previous attempts to prevent IP address assignment conflicts on the affected ESX host(s)

esxcli network ip interface remove -i <vmk_name>
esxcli network vswitch standard remove -v <vswitch_name>
Since it is a multi-DVS configuration with separate DVS for storage, create a separate NFS standard virtual switch on the ESX host currently hosting the VCF installer VM with the same name as per the separate DVS for Storage

esxcli network vswitch standard add --vswitch-name=<NFS_DVS_name>
Associate one of the physical NICs defined in the input dvsSpec to the newly created standard switch.

esxcli network vswitch standard uplink add --uplink-name=<vmnic_id> --vswitch-name=<NFS_DVS_name>
Unmount the NFS datastore and remove the old incorrectly configured NFS port group along with its vmkernel adapter :

esxcli storage nfs remove -v <NFS_Datastore_Name>
esxcli network ip interface remove -i <vmk_NFS_old_name> esxcli network vswitch standard portgroup remove --portgroup-name=<NFS_old_pg_name> --vswitch-name=<vswitch_name>
Create the NFS port group under the new standard switch, ensuring the name exactly matches the portGroup defined in the input network specification for NFS
esxcli network vswitch standard portgroup add --portgroup-name=<NFS_DVS_pg_name> --vswitch-name=<NFS_DVS_name>
(Optional)Configure VLAN if needed
esxcli network vswitch standard portgroup set -p <NFS_DVS_pg_name> --vlan-id <vlan_id>
Assign the NFS IP address from the designated NFS IP range provided in the input specification
Remount the NFS datastore

Once the above configurations have been verified, proceed to retry or re-attempt the deployment task

Additional Information

Steps to unmount and mount NFS datastore : Remounting a disconnected NFS datastore from the ESXi command line