Virtual machine PXE boot fails when IP fragmentation is used

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

A virtual machine might fail to complete PXE boot when it is configured with BIOS firmware and the network environment requires the use of IP fragmentation during the TFTP transfer of the Network Bootstrap Program (NBP).  The virtual machine completes DHCP and performs the initial TFTP request, but will not respond to fragmented TFTP DATA packets.  After approximately nine minutes, the TFTP transfer times out and the PXE boot fails, displaying the following messages on the Virtual Machine's screen:
PXE-E32: TFTP open timeout
PXE-E32: TFTP open timeout
PXE-M0F: Exiting Intel PXE ROM.

Environment

VMware vSphere ESXi 7.0.2

Cause

The BIOS PXE Base Code contains a minimal IPv4 stack which is limited in its ability to handle complex scenarios, including scenarios which might be considered trivial within a more regular IPv4 implementation.  It operates in a constrained environment with only a small amount of memory available for its use.

The BIOS PXE Base Code is capable of recognizing and reassembling fragmented IPv4 packets, however that capability is only available when sufficient packet buffer memory is available for reassembling the packet.

• IPv4 fragment reassembly is not available during TFTP transfers managed directly by the BIOS PXE Base Code, including the initial fetch of the NBP and any TFTP transfers subsequently performed using the PXE TFTP API.  Any incoming fragmented IPv4 packets will be silently dropped.

• IPv4 fragment reassembly will be automatically performed when the running NBP uses the PXE UDP API and supplies a buffer adequate for receiving the reassembled packet.  If an incoming fragmented IPv4 packet is too large for the given buffer, it will be silently dropped.

• As per the design of the PXE UNDI API, an NBP which directly uses the UNDI interface will not use the PXE Base Code IPv4 stack and must perform its own IP fragment reassembly if required.

Resolution

Currently there is no resolution to the issue.

Workaround:

To workaround the issue, please follow any one of the two potential workarounds available:

Option 1: Adjust the network path MTU or the TFTP block size to eliminate the need for IP fragmentation during BIOS PXE TFTP transfers.

-OR-

Option 2: Consider using EFI firmware.  The limitations on memory usable for IP fragment reassembly are specific to BIOS PXE and do not affect EFI PXE.