VMs lost network connectivity for a few minutes after being migrated to an empty host via vMotion.

Products

• VMware NSX

Issue/Introduction

From ESXi Host,

It is observed in /var/run/log/vmkernel.log that bringing BFD sessions up takes longer than expected—possibly several minutes.

<DATE_TIME> cpu31:2105385)BFD_CreateSession:392:[nsx@6876 comp="nsx-esx" subcomp="bfd"]session: 0x58e2f4ea, local: <HOST_TEP_IP>, remote: <EDGE_TEP_IP>, type: overlay ... <DATE_TIME> cpu63:2105111)BFD_HandleStatusChange:709:[nsx@6876 comp="nsx-esx" subcomp="bfd"]local: <HOST_TEP_IP>, remote: <EDGE_TEP_IP>, oldState: init, newState: up, diag: No Diagnostic, type: overlay

 

Meanwhile, it is seen in vmware.log and /var/run/log/vmkernel.log that the vMotion completes and the VM's vNIC reconnects to the virtual switch.

vmware.log <DATE_TIME> In(05) vcpu-0 - MigrateSetState: Transitioning from state 12 to 0.   vmkernel.log <DATE_TIME> cpu46:179630791)Net: 2184: connected <VM_NAME>.eth0 eth0 to vDS, portID <PORT_ID> <DATE_TIME> cpu46:179630791)NetPort: 1543: enabled port <PORT_ID> with mac <MAC_ADDRESS>

 

It might be detected in /var/run/log/dfwpktlogs.log that ICMP to <VM_IP> fails due to unreachable reason code.

<DATE_TIME> <HOSTNAME> FIREWALL-PKTLOG: 48921027 INET match PASS 2046 IN 80 ICMP 3 1 <GATEWAY_IP>-><VM_IP> ... <DATE_TIME> <HOSTNAME> FIREWALL-PKTLOG: 48921027 INET match PASS 2046 IN 88 ICMP 3 1 <GATEWAY_IP>-><VM_IP>

 

From Edge,

It is observed in /var/log/syslog that the firewall reconfiguration is being applied, and the process appears to be taking longer than expected to complete.

<DATE_TIME> <HOSTNAME> NSX 8310 FIREWALL [nsx@6876 comp=""nsx-edge"" subcomp=""datapathd"" s2comp=""firewall"" tname=""dp-ipc55"" level=""INFO""] Firewall apply total: <DURATION> msec ... <DATE_TIME> <HOSTNAME> NSX 8310 FIREWALL [nsx@6876 comp=""nsx-edge"" subcomp=""datapathd"" s2comp=""firewall"" tname=""dp-ipc55"" level=""INFO""] Firewall apply total: <DURATION> msec

 

Environment

VMware NSX

Cause

Bringing BFD sessions up might take longer than expected, as the Edge node is quite busy applying the firewall configuration.

Resolution

As a workaround:

Workaround is to have a dummy VM on the host for the BFD tunnels to remain UP.

 

In NSX 4.2, performance improvement is introduced for applying firewall configurations.