"ESX unrecoverable error: (vcpu-0) vcpu-0: EPT misconfiguration: PA fb100000" error seen in crashing virtual machines

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Multiple virtual machines continually crash.
• In the virtual machine events, you see the messages similar to:

ESX unrecoverable error: (vcpu-x) vcpu-x: EPT misconfiguration: PA fb100000
 

• In the /vmfs/volumes/datastore/vm_name/vmware.log, you see the entries similar to:

 

2018-06-29T05:19:54.254Z| vcpu-1| W115: MONITOR PANIC: vcpu-4:EPT misconfiguration: PA 3f552b410

2018-06-29T05:19:54.255Z| vcpu-1| I125: Core dump with build build-8294253
2018-06-29T05:19:54.255Z| vcpu-7| I125: Exiting vcpu-7
2018-06-29T05:19:54.255Z| vcpu-3| I125: Exiting vcpu-3
2018-06-29T05:19:54.255Z| vcpu-5| I125: Exiting vcpu-5
2018-06-29T05:19:54.255Z| vcpu-2| I125: Exiting vcpu-2
2018-06-29T05:19:54.255Z| vcpu-0| I125: Exiting vcpu-0
2018-06-29T05:19:54.255Z| vcpu-4| I125: Exiting vcpu-4
2018-06-29T05:19:54.255Z| vcpu-6| I125: Exiting vcpu-6
2018-06-29T05:19:54.264Z| vcpu-1| I125: Writing monitor file `vmmcores.gz`
2018-06-29T05:19:54.277Z| vcpu-1| W115: Dumping core for vcpu-0

2018-06-29T05:20:06.359Z| vcpu-1| I125: Msg_Post: Error
2018-06-29T05:20:06.359Z| vcpu-1| I125: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-1)
2018-06-29T05:20:06.359Z| vcpu-1| I125+ vcpu-4:EPT misconfiguration: PA 3f552b410
2018-06-29T05:20:06.359Z| vcpu-1| I125: [msg.panic.haveLog] A log file is available in "/vmfs/volumes/58b5c2e7-f57129c8-493e-0025b506b300/USLILVMPSAPP20/vmware.log".
2018-06-29T05:20:06.359Z| vcpu-1| I125: [msg.panic.requestSupport.withoutLog] You can request support.
2018-06-29T05:20:06.359Z| vcpu-1| I125: [msg.panic.requestSupport.vmSupport.vmx86]
2018-06-29T05:20:06.359Z| vcpu-1| I125+ To collect data to submit to VMware technical support, run "vm-support".
2018-06-29T05:20:06.359Z| vcpu-1| I125: [msg.panic.response] We will respond on the basis of your support entitlement.
2018-06-29T05:20:06.359Z| vcpu-1| I125: ----------------------------------------

• Running grep -i "misconfiguration: PA" /var/run/log/vmkernel.log on the ESXi host returns similar to:

2018-06-27T02:40:50.098Z cpu52:71994)WARNING: World: vm 71994: 8725: vmm0:VM1:vcpu-0:EPT misconfiguration: PA 1a0a038
2018-06-27T02:51:40.947Z cpu52:73968)WARNING: World: vm 73968: 8725: vmm0:VM2:vcpu-0:EPT misconfiguration: PA 1e2755030
2018-06-29T05:19:54.254Z cpu52:70087)WARNING: World: vm 70087: 8725: vmm4:vM3:vcpu-4:EPT misconfiguration: PA 3f552b410
2018-06-26T18:57:54.442Z cpu52:70593)WARNING: World: vm 70593: 8725: vmm3:VM4:vcpu-3:EPT misconfiguration: PA 309f1b020
2018-06-27T01:31:53.554Z cpu52:74631)WARNING: World: vm 74631: 8725: vmm2:VM5:vcpu-2:EPT misconfiguration: PA 5485f7938
2018-06-26T18:12:30.459Z cpu52:3538220)WARNING: World: vm 3538220: 12387: vmm3:VM6:vcpu-3:EPT misconfiguration: PA 33b088146
2018-06-26T18:20:56.409Z cpu52:3538760)WARNING: World: vm 3538760: 12387: vmm1:VM7:vcpu-1:EPT misconfiguration: PA fcfacdd00
2018-06-24T01:24:56.400Z cpu52:1167230)WARNING: World: vm 1167230: 12387: vmm4:VM7:vcpu-4:EPT misconfiguration: PA 1654778ef0
2018-06-24T05:43:40.534Z cpu52:3362235)WARNING: World: vm 3362235: 12387: vmm6:VM7:vcpu-6:EPT misconfiguration: PA 1738a00c80
2018-06-25T11:17:20.018Z cpu52:3452793)WARNING: World: vm 3452793: 12387: vmm3:VM8:vcpu-3:EPT misconfiguration: PA 36eb36b00
2018-05-20T11:00:00.913Z cpu52:694467)WARNING: World: vm 694467: 12387: vmm3:VM7:vcpu-3:EPT misconfiguration: PA 264c000
2018-05-27T10:17:59.771Z cpu52:1606260)WARNING: World: vm 1606260: 12387: vmm0:VM9:vcpu-0:EPT misconfiguration: PA bffb1b90
2018-05-02T14:59:06.504Z cpu52:1611493)WARNING: World: vm 1611493: 12387: vmm1:vM10:vcpu-1:EPT misconfiguration: PA 79dec068
2018-05-02T19:58:08.458Z cpu52:42658)WARNING: World: vm 42658: 12387: vmm2:VM11:vcpu-2:EPT misconfiguration: PA 3ad8b3850
2018-05-03T00:25:09.650Z cpu52:56551)WARNING: World: vm 56551: 12387: vmm7:VM12:vcpu-7:EPT misconfiguration: PA 288302c90
2018-05-02T11:36:50.084Z cpu52:43081)WARNING: World: vm 43081: 12387: vmm0:VM13:vcpu-0:EPT misconfiguration: PA 5451e0

Notes:

• All the failures are occurring on cpu52 in this example indicating an issue with the physical CPU.
• These entries are aggregated over several vmkernel.log files.

Resolution

Confirm the BIOS, Firmware,and processor microcode are all up-to-date. If the issue persists the CPU will need to be replaced by the hardware vendor.

Note: Swapping the CPU between sockets and reinstall the memory to isolate the faulting CPU.