Describe the behavior of enabling Provisioning service (the checkbox listed in vmkernel adapter settings) in various configurations.
The Provisioning service uses NFC during cold migrations, cloning, and snapshot creation.
Cold migrations are sometimes needed for migrating VM's between physical datacenters, possibly over a WAN, where the source and destination hosts do not have access to the same datastores.
Environment
VMware vCenter Server 6.x VMware vSphere ESXi 6.x VMware vCenter Server 7.x VMware vSphere ESXi 7.x VMware vCenter Server 8.x VMware vSphere ESXi 8.x
Resolution
Scenario 1:
No Provisioning vmk's are selected
Result:
Source host uses vmk0
Destination host uses vmk0
Scenario 2:
Both Provisioning vmk's are on same subnet (no routing required)
Result:
Source host uses Provisioning vmkX
Destination host uses Provisioning vmkX
Scenario 3:
Provisioning vmk's are on different subnets
ESXi static routes exist
Physical routes between subnets exists
Result:
Source host uses Provisioning vmkX
Destination host uses Provisioning vmkX
Scenario 4:
Provisioning vmk's are on different subnets
ESXi static routes does NOT exist
Physical routes between subnets exists
Result:
Source host uses vmk0
Destination host uses Provisioning vmkX
Scenario 5:
Provisioning vmk's are on different subnets
ESXi static routes do NOT exist
Physical routes between subnets do NOT exist
Result:
Migration/vMotion fails at 23%
"Cannot connect to host" error reported on vMotion failure.
Additional Information
TCP port 902 must be open between Source and Destination hosts for NFC traffic used during cold migrations. If the VM is powered up, then all of the migration traffic will traverse the vmk assigned to the vMotion service using TCP port 8000.
vSphere 8.0 introduces UDT (Unified Data Transport) which enhances cold migration by using NFC as a control channel but offloads the data transfer to vSphere vMotion protocol. When configured by enabling provisioning vmk, control traffic would go through port 902, while data traffic would go through TCP port 8000. For more info see: vSphere vMotion Unified DataTransport