VMware ESXi 5.5.x and 6.x hosts experiences a purple diagnostic screen mentioning ALERT: NMI: 709: NMI IPI received

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The ESXi 5.5.x and 6.x hosts fails with a purple diagnostic screen.
• The purple diagnostic screen contains entries similar to:
  
  @BlueScreen: #PF Exception 14 in world wwww:WorldName IP 0xnnnnnnnn addr 0x0
  PTEs:0xnnnnnnnn;0xnnnnnnnn;0x0;
  0xnnnnnnnn:[0xnnnnnnnn]PFrame_SetPinCount@vmkernel#nover+0x3
  0xnnnnnnnn:[0xnnnnnnnn]VmMemPin_GetMainMemRange@vmkernel#nover+0x36c
  0xnnnnnnnn:[0xnnnnnnnn]P2MCache_Get@vmkernel#nover+0x120
  0xnnnnnnnn:[0xnnnnnnnn]P2MCache_GetDirectPA@vmkernel#nover+0x24
  0xnnnnnnnn:[0xnnnnnnnn]Vmxnet3VMKDevMapRxBufferChunk@vmkernel#nover+0x6e
  0xnnnnnnnn:[0xnnnnnnnn]Vmxnet3VMKDevRxWithLock@vmkernel#nover+0x5a5
  0xnnnnnnnn:[0xnnnnnnnn]Vmxnet3VMKDevRx@vmkernel#nover+0x2da
  0xnnnnnnnn:[0xnnnnnnnn]IOChain_Resume@vmkernel#nover+0x210
  0xnnnnnnnn:[0xnnnnnnnn]PortOutput@vmkernel#nover+0xae
  0xnnnnnnnn:[0xnnnnnnnn][email protected]#v2_3_0_0+0x3e9
  0xnnnnnnnn:[0xnnnnnnnn]VSIPDVFProcessSlowPathPackets@<None>#<None>+0x16

Note: For additional symptoms, see the Additional Information section.

Cause

This issue occurs when a large number of packets are transmitted/received by a service VM. The DVFilter continues to dominate the CPU resulting in a heartbeat lost and a purple diagnostic screen.

Resolution

To resolve this issue, upgrade the ESXi host to any of the following ESXi versions that are the minimum required to use Distributed Service Insertion.

• ESXi 5.5 Patch 10 (Build 4722766), available at VMware Downloads.

• ESXi 6.0 Update 3, available at VMware Downloads.

• ESXi 6.5, available at VMware Downloads.

Additional Information

You see these additional symptoms:

• In the var/log/vmkernel.log file of the affected ESXi host, you see entries similar to:
  
  2015-06-30T10:40:07.522Z cpu0:33372)DVFilter: 917: OOM or retry on 30 bytes long message. Trying again later
  2015-06-30T10:40:16.806Z cpu1:33372)vsip VSIPDVFSetSlowPathReply:287: Slow path reply for filter attach : Success
  2015-06-30T10:40:42.522Z cpu1:33372)DVFilter: 311: Retry on 30 bytes long message. Trying again later
  2015-06-30T10:40:57.334Z cpu12:277957)VSCSIFs: 2235: handle 8193(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:Palo_Alto_Networks_NGFW_(3))
  2015-06-30T10:40:57.390Z cpu12:277957)VSCSIFs: 2235: handle 8193(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:Palo_Alto_Networks_NGFW_(3))
  2015-06-30T10:40:57.447Z cpu12:277957)VSCSIFs: 2235: handle 8193(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:Palo_Alto_Networks_NGFW_(3))
  2015-06-30T10:40:57.447Z cpu12:277957)VSCSIFs: 2235: handle 8193(vscsi0:0):Invalid Opcode (0x37) from (vmm0:Palo_Alto_Networks_NGFW_(3))
  ^[[7m2015-06-30T10:41:10.521Z cpu23:33013)WARNING: Heartbeat: 781: PCPU 0 didn't have a heartbeat for 8 seconds; *may* be locked up.^[[0m
  ^[[31;1m2015-06-30T10:41:10.522Z cpu0:289037)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0x65171(0x41802b200000):0x43940869bdd4:0x4010](Src 0x1, CPU0)^[[0m
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.