BFD Session Flaps and Network Disruption After vMotion of a VM with a Large Distributed Firewall IP Address Set.

Products

• VMware vDefend Firewall

Issue/Introduction

When a Virtual Machine (VM) protected by NSX Distributed Firewall with a very large number of IP address group members is live-migrated (vMotion) to a new ESXi host, the destination host may experience transient network disruption immediately following the migration. This can manifest as BFD session flaps, brief packet loss, or in severe cases, a Vmxnet3 hang on the destination host.

 

Symptoms:

 

The following log Symptoms indicate this condition is occurring:

 

• The destination ESXi host vmkernel.log shows a large IP address table being restored during vMotion:
  (Note: In the below log example an 'ImportStateTLV entry type 1' payload translates roughly to 1.7 Million IP addresses being loaded into memory concurrently (27466191/16 = ~ 1,7 M IPs))

 

vmkernel: cpu[CPU_ID]:[WORLD_ID])Restore state called for filter [FILTER_NAME].2

vmkernel: cpu[CPU_ID]:[WORLD_ID])Importing [FILTER_NAME].2, Version 1100

vmkernel: cpu[CPU_ID]:[WORLD_ID])VSIP module ioctls: disabled

vmkernel: cpu[CPU_ID]:[WORLD_ID])ImportStateTLV entry type 12, len 52, cnt 1

vmkernel: cpu[CPU_ID]:[WORLD_ID])Importing from source version RELEASEbuild-[BUILD_NUMBER]

vmkernel: cpu[CPU_ID]:[WORLD_ID])ImportStateTLV entry type 1, len 27466191, cnt 1179

vmkernel: cpu[CPU_ID]:[WORLD_ID])ImportStateTLV entry type 2, len 1033427, cnt 2598

vmkernel: cpu[CPU_ID]:[WORLD_ID])Importing succeeded

 

• BFD sessions on the destination host's TEP interfaces flap.

BFD_HandleStatusChange: local: [LOCAL_IP], remote: [REMOTE_IP], oldState: up, newState: down, diag: Control Detection Time Expired

 

• In more severe cases, a PCPU heartbeat warning or Vmxnet3 hang may also appear in the logs.

 

Example log snippet for PCPU heartbeat warnings:

WARNING: Heartbeat: 961: PCPU [PCPU_ID] didn't have a heartbeat for 6 seconds, timeout is 10; *may* be locked up.

ALERT: NMI: 738: NMI IPI: PC [MEMORY_ADDR] (Src [SRC_ID], CPU[PCPU_ID]) [e-mail] [e-mail] [e-mail]

 

Example log snipped forVmxnet3 hang:

Vmxnet3: Hang detected, numHangQ: 2, enableGen: [GEN_ID]

Vmxnet3: portID:[PORT_ID], QID: 2, ringSize: 512 inFlight: 2, delay(ms): 10006

 

 

Environment

NSX 3.x, 4.x, 9.x
ESXi hosts configured with NSX Distributed Firewall
VMs associated with large IP address groups (approaching or exceeding scale maximums)

 

Cause

During vMotion, the NSX data path module on the destination host must restore the migrated VM's complete firewall policy state, including all associated IP address group members, before the VM's network is re-enabled. When the IP address set is very large (hundreds of thousands to millions of entries), this restoration process can consume a single CPU core for several seconds without interruption. During this window, time-sensitive network operations on that host, such as BFD keepalive packet processing, are delayed long enough to trigger session timeouts.

Resolution

Ensure that IP address group membership stays within validated NSX scale limits. Refer to the VMware Configuration Maximums guide for the current supported limits per NSX version.