"ESX unrecoverable error: (vcpu-0) vcpu-0: EPT misconfiguration: PA fb100000" error seen in crashing virtual machines

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple virtual machines continually crash.
• In the virtual machine events, you see the messages similar to:

ESX unrecoverable error: (vcpu-x) vcpu-x: EPT misconfiguration: PA fb100000

• In the /vmfs/volumes/datastore/vm_name/vmware.log, you see the entries similar to:

YYYY-MM-DDTHH:MM:S| vcpu-1| W115: MONITOR PANIC: vcpu-4:EPT misconfiguration: PA 3f552b410
YYYY-MM-DDTHH:MM:S.255Z| vcpu-1| I125: Core dump with build build-8294253
YYYY-MM-DDTHH:MM:S.255Z| vcpu-7| I125: Exiting vcpu-7
YYYY-MM-DDTHH:MM:S.255Z| vcpu-3| I125: Exiting vcpu-3
YYYY-MM-DDTHH:MM:S.255Z| vcpu-5| I125: Exiting vcpu-5
YYYY-MM-DDTHH:MM:S.255Z| vcpu-2| I125: Exiting vcpu-2
YYYY-MM-DDTHH:MM:S.255Z| vcpu-0| I125: Exiting vcpu-0
YYYY-MM-DDTHH:MM:S.255Z| vcpu-4| I125: Exiting vcpu-4
YYYY-MM-DDTHH:MM:S.255Z| vcpu-6| I125: Exiting vcpu-6
YYYY-MM-DDTHH:MM:S.264Z| vcpu-1| I125: Writing monitor file `vmmcores.gz`
YYYY-MM-DDTHH:MM:S.277Z| vcpu-1| W115: Dumping core for vcpu-0

YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: Msg_Post: Error
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-1)
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125+ vcpu-4:EPT misconfiguration: PA 3f552b410
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: [msg.panic.haveLog] A log file is available in "/vmfs/volumes/datastore/vm_name/vmware.log".
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: [msg.panic.requestSupport.withoutLog] You can request support.
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: [msg.panic.requestSupport.vmSupport.vmx86]
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125+ To collect data to submit to VMware technical support, run "vm-support".
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: [msg.panic.response] We will respond on the basis of your support entitlement.
YYYY-MM-DDTHH:MM:S.359Z| vcpu-1| I125: ----------------------------------------

• Running grep -i "misconfiguration: PA" /var/run/log/vmkernel.log on the ESXi host returns similar to:

YYYY-MM-DDTHH:MM:S.098Z cpu52:71994)WARNING: World: vm 71994: 8725: vmm0:VM1:vcpu-0:EPT misconfiguration: PA 1a0a038
YYYY-MM-DDTHH:MM:S.947Z cpu52:73968)WARNING: World: vm 73968: 8725: vmm0:VM2:vcpu-0:EPT misconfiguration: PA 1e2755030
YYYY-MM-DDTHH:MM:S254Z cpu52:70087)WARNING: World: vm 70087: 8725: vmm4:vM3:vcpu-4:EPT misconfiguration: PA 3f552b410
YYYY-MM-DDTHH:MM:S.442Z cpu52:70593)WARNING: World: vm 70593: 8725: vmm3:VM4:vcpu-3:EPT misconfiguration: PA 309f1b020
YYYY-MM-DDTHH:MM:S.554Z cpu52:74631)WARNING: World: vm 74631: 8725: vmm2:VM5:vcpu-2:EPT misconfiguration: PA 5485f7938
YYYY-MM-DDTHH:MM:S.459Z cpu52:3538220)WARNING: World: vm 3538220: 12387: vmm3:VM6:vcpu-3:EPT misconfiguration: PA 33b088146
YYYY-MM-DDTHH:MM:S.409Z cpu52:3538760)WARNING: World: vm 3538760: 12387: vmm1:VM7:vcpu-1:EPT misconfiguration: PA fcfacdd00
YYYY-MM-DDTHH:MM:S.400Z cpu52:1167230)WARNING: World: vm 1167230: 12387: vmm4:VM7:vcpu-4:EPT misconfiguration: PA 1654778ef0
YYYY-MM-DDTHH:MM:S.534Z cpu52:3362235)WARNING: World: vm 3362235: 12387: vmm6:VM7:vcpu-6:EPT misconfiguration: PA 1738a00c80
YYYY-MM-DDTHH:MM:S.018Z cpu52:3452793)WARNING: World: vm 3452793: 12387: vmm3:VM8:vcpu-3:EPT misconfiguration: PA 36eb36b00
YYYY-MM-DDTHH:MM:S.913Z cpu52:694467)WARNING: World: vm 694467: 12387: vmm3:VM7:vcpu-3:EPT misconfiguration: PA 264c000
YYYY-MM-DDTHH:MM:S.771Z cpu52:1606260)WARNING: World: vm 1606260: 12387: vmm0:VM9:vcpu-0:EPT misconfiguration: PA bffb1b90
YYYY-MM-DDTHH:MM:S.504Z cpu52:1611493)WARNING: World: vm 1611493: 12387: vmm1:vM10:vcpu-1:EPT misconfiguration: PA 79dec068
YYYY-MM-DDTHH:MM:S.458Z cpu52:42658)WARNING: World: vm 42658: 12387: vmm2:VM11:vcpu-2:EPT misconfiguration: PA 3ad8b3850
YYYY-MM-DDTHH:MM:S.650Z cpu52:56551)WARNING: World: vm 56551: 12387: vmm7:VM12:vcpu-7:EPT misconfiguration: PA 288302c90
YYYY-MM-DDTHH:MM:S.084Z cpu52:43081)WARNING: World: vm 43081: 12387: vmm0:VM13:vcpu-0:EPT misconfiguration: PA 5451e0

Notes:

• All the failures are occurring on cpu52 in this example indicating an issue with the physical CPU.
• These entries are aggregated over several vmkernel.log files.

Environment

VMware vSphere ESXI 7.x

VMware vSphere ESXI 8.x

Resolution

Confirm the BIOS, Firmware,and processor microcode are all up-to-date. If the issue persists the CPU will need to be replaced by the hardware vendor.

Note: Swapping the CPU between sockets and reinstall the memory to isolate the faulting CPU.