vSphere Provisioning Service Behavior

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Describe the behavior of enabling Provisioning service (the checkbox listed in vmkernel adapter settings) in various configurations.
• The Provisioning service uses NFC during cold migrations, cloning, and snapshot creation.
• Cold migrations are sometimes needed for migrating VM's between physical datacenters, possibly over a WAN, where the source and destination hosts do not have access to the same datastores.
  
  

Environment

VMware vCenter Server 6.x
VMware vSphere ESXi 6.x
VMware vCenter Server 7.x
VMware vSphere ESXi 7.x
VMware vCenter Server 8.x
VMware vSphere ESXi 8.x

Resolution

Scenario 1:

• No Provisioning vmk's are selected

Result:

• Source host uses vmk0
• Destination host uses vmk0

Scenario 2:

• Both Provisioning vmk's are on same subnet (no routing required)

Result:

• Source host uses Provisioning vmkX
• Destination host uses Provisioning vmkX

Scenario 3:

• Provisioning vmk's are on different subnets
• ESXi static routes exist
• Physical routes between subnets exists

Result:

• Source host uses Provisioning vmkX
• Destination host uses Provisioning vmkX

Scenario 4:

• Provisioning vmk's are on different subnets
• ESXi static routes does NOT exist
• Physical routes between subnets exists

Result:

• Source host uses vmk0
• Destination host uses Provisioning vmkX

Scenario 5:

• Provisioning vmk's are on different subnets
• ESXi static routes do NOT exist
• Physical routes between subnets do NOT exist

Result:

• Migration/vMotion fails at 23%
• "Cannot connect to host" error reported on vMotion failure.

Additional Information

TCP port 902 must be open between Source and Destination hosts for NFC traffic used during cold migrations. If the VM is powered up, then all of the migration traffic will traverse the vmk assigned to the vMotion service using TCP port 8000.

ESXi static routes: Configuring static routes for vmkernel ports on an ESXi host

vSphere 8.0 introduces UDT (Unified Data Transport) which enhances cold migration by using NFC as a control channel but offloads the data transfer to vSphere vMotion protocol. When configured by enabling provisioning vmk, control traffic would go through port 902, while data traffic would go through TCP port 8000. For more info see: vSphere vMotion Unified DataTransport