Cloning template via provisioning network fails after upgrade to ESXi 8.0.

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

Issue
      After upgrading ESXI Host from 7.x to 8.x, the cloning of (remote) templates or VMs fails. The templates would usually be cloned via a provisioning network. Logs confirm that files could not be accessed for copying.

Conditions to trigger the issue

1. Both involved ESXi hosts have a VMK enabled for provisioning.
2. The network in between is firewalled and configured for allowing port 902/TCP (aka NFC port / Network File Copy), only. For vSphere 7 this port was sufficient.
3. The clone process cannot make use of a shared drive on either side.

Environment

vSphere ESXi 8.x

Cause

UDT conflicts with firewall configuration

With vSphere 8 a feature called UDT (Universal Data Transport) was introduced. It extends the NFC protocol, rsp. merges a part of the vMotion protocol into it, if and only if a provisioning network connection (VMK with Provisioning enabled on them) is used. The transfer of files using the vMotion protocol is usually by magnitudes faster than the classic NFC.

In such a this case, port 902/TCP remains in use only for preparation of the file copies, but port 8000/TCP of the vMotion protocol is used to transfer the payload, i.e. the mere files' content. And this communication does run on the provisioning network and not via an eventually configured vMotion network.

Now, if a firewall is in use on the hardware of the provisioning network, so in between the ESXi hosts, for version 7 it would only need allow 902/TCP. When UDT comes to play since version 8.0, port 8000/TCP is required additionally on this network.

Failure to allow port 8000/TCP would result in file copy failures. Regrettably, the ESXi will not alert a lost connection as 902/TCP works. Also, the logs would likely not show prominent network errors but file-access errors.

Resolution

Workaround

A simple and fast workaround would be to disable/delete the provisioning-flag from the VMK on one or all hosts. Having no provisioning network on at least either of the two involved hosts disables the UDT feature und fails back to the classic NFC via 902/TCP also transferring the payloads via 902/TCP.

Using this workaround, the provisioning network is effectively disabled and the management-network will be used for the copy, so a drastically higher load on the management-traffic will likely be a side-effect as long as the workaround is in use.

Solution

Best practices suggest to not use firewalling between ESXi hosts on networks connecting VMKs at all. Now, reality often dictates a higher level of security. If a firewall is obligatory, then for provisioning networks enable both ports in both directions:

1. 902/TCP (NFC), bidirectionally
2. 8000/TCP (vMotion), bidirectionally

Usually in larger organizations, firewall-changes take some time. In this case temporarily disable the provisioning network (see "Workaround" above) until the firewalls are updated.

Additional Information

More on UDT

• Documentation of UDT