ESXi 6.x crashes with a PSOD - PF Exception 14 in world 65688:HELPER_IMMED IP
Article ID: 318671
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
To avoid the PSOD.
Symptoms:
0x439144c1baf0:[0x4180002b4cca]PCIConfig_ReadSafe@vmkernel#nover+0x7a stack: 0x0, 0x439144c1bca0, 0x4301ad0bdc50, 0x439144c1bca0, 0xaa
0x439144c1bb40:[0x4180002a73de]PCIConfig_Read@vmkernel#nover+0xe stack: 0x0, 0x418000093855, 0x4301ad4c1e30, 0x418000396fa2, 0x800000000
0x439144c1bb60:[0x418000093855]AcpiOsReadPciConfiguration@vmkernel#nover+0x75 stack: 0x800000000, 0x4767613e, 0x41804f400000, 0x439141ea7280, 0x439144c27100
0x439144c1bb70:[0x418000396fa2]AcpiEvAddressSpaceDispatch@vmkernel#nover+0x11a stack: 0x41804f400000, 0x439141ea7280, 0x439144c27100, 0x0, 0x1
0x439144c1bbe0:[0x41800039a6b4]AcpiExAccessRegion@vmkernel#nover+0x188 stack: 0x4301ad063590, 0x439144c1bca0, 0x4301ad063590, 0x4301ad13e950, 0x439144c1bca0
0x439144c1bc40:[0x41800039aacc]AcpiExFieldDatumIo@vmkernel#nover+0xa0 stack: 0x41800049a1d8, 0x4180003ae0fc, 0x7, 0x0, 0x8
0x439144c1bc80:[0x41800039ad3d]AcpiExExtractFromField@vmkernel#nover+0xa9 stack: 0x4301ad4c1f30, 0x0, 0x0, 0x4180003ae31c, 0x0
0x439144c1bcf0:[0x41800039a235]AcpiExReadDataFromField@vmkernel#nover+0xa5 stack: 0x4301ad13e950, 0x4301ad4c0b50, 0x4301ad2ea830, 0x41800039dbb6, 0x4301ad063590
0x439144c1bd20:[0x41800039dbb6]AcpiExResolveNodeToValue@vmkernel#nover+0xc6 stack: 0x4301ad4c0b50, 0x0, 0xb, 0x4301ad4c0e30, 0x4301ad0be3a0
0x439144c1bd70:[0x41800039dde7]AcpiExResolveToValue@vmkernel#nover+0x67 stack: 0x4301ad4c1e10, 0x4301ad0be3a0, 0x4301ad4c0b50, 0x4301ad2ea830, 0x11
0x439144c1bdb0:[0x418000393a60]AcpiDsEvaluateNamePath@vmkernel#nover+0x94 stack: 0x4301ad4c0b50, 0x0, 0x4301ad2ea830, 0x418000394163, 0x4301ad42ae80
0x439144c1bde0:[0x418000394163]AcpiDsExecEndOp@vmkernel#nover+0x39b stack: 0x4301ad4c0b50, 0x4301ad4c0b50, 0x0, 0x410016626855, 0x4301ad4c0b80
0x439144c1be20:[0x4180003a5f82]AcpiPsParseLoop@vmkernel#nover+0xd2 stack: 0x4301ad2ea830, 0x4301ad2ea830, 0x0, 0x0, 0x430100000003
0x439144c1be90:[0x4180003a6eb9]AcpiPsParseAml@vmkernel#nover+0x219 stack: 0x4301ad4c0f80, 0x4301ad464468, 0x0, 0x4180003a75cd, 0x430600000003
0x439144c1bec0:[0x4180003a75cd]AcpiPsExecuteMethod@vmkernel#nover+0x125 stack: 0x4301ad464450, 0x4301ad464470, 0x4301ad0f5f00, 0x0, 0x430614ff5640
0x439144c1bf00:[0x4180003a17b5]AcpiNsEvaluate@vmkernel#nover+0x15d stack: 0x4301ad4bdcf0, 0x4301ad464450, 0x430614ff57c0, 0x418000395ff2, 0x430614ff5080
0x439144c1bf30:[0x418000395ff2]AcpiEvAsynchExecuteGpeMethod@vmkernel#nover+0x112 stack: 0x430614ff57c0, 0x4180000ca3ed, 0x4307ad8b3050, 0x0, 0x4301c0b31050
0x439144c1bf50:[0x4180000ca3ed]helpFunc@vmkernel#nover+0x3c5
0x439144c1bfe0:[0x4180002cb675]CpuSched_StartWorld@vmkernel#nover+0x99 stack: 0x0, 0x0, 0x0, 0x0, 0x0
Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Symptoms:
- ESXi host running on 6.x crashes with a PSOD - PF Exception 14 in world 65688:HELPER_IMMED IP
- PSOD backtrace will show similar to below
0x439144c1baf0:[0x4180002b4cca]PCIConfig_ReadSafe@vmkernel#nover+0x7a stack: 0x0, 0x439144c1bca0, 0x4301ad0bdc50, 0x439144c1bca0, 0xaa
0x439144c1bb40:[0x4180002a73de]PCIConfig_Read@vmkernel#nover+0xe stack: 0x0, 0x418000093855, 0x4301ad4c1e30, 0x418000396fa2, 0x800000000
0x439144c1bb60:[0x418000093855]AcpiOsReadPciConfiguration@vmkernel#nover+0x75 stack: 0x800000000, 0x4767613e, 0x41804f400000, 0x439141ea7280, 0x439144c27100
0x439144c1bb70:[0x418000396fa2]AcpiEvAddressSpaceDispatch@vmkernel#nover+0x11a stack: 0x41804f400000, 0x439141ea7280, 0x439144c27100, 0x0, 0x1
0x439144c1bbe0:[0x41800039a6b4]AcpiExAccessRegion@vmkernel#nover+0x188 stack: 0x4301ad063590, 0x439144c1bca0, 0x4301ad063590, 0x4301ad13e950, 0x439144c1bca0
0x439144c1bc40:[0x41800039aacc]AcpiExFieldDatumIo@vmkernel#nover+0xa0 stack: 0x41800049a1d8, 0x4180003ae0fc, 0x7, 0x0, 0x8
0x439144c1bc80:[0x41800039ad3d]AcpiExExtractFromField@vmkernel#nover+0xa9 stack: 0x4301ad4c1f30, 0x0, 0x0, 0x4180003ae31c, 0x0
0x439144c1bcf0:[0x41800039a235]AcpiExReadDataFromField@vmkernel#nover+0xa5 stack: 0x4301ad13e950, 0x4301ad4c0b50, 0x4301ad2ea830, 0x41800039dbb6, 0x4301ad063590
0x439144c1bd20:[0x41800039dbb6]AcpiExResolveNodeToValue@vmkernel#nover+0xc6 stack: 0x4301ad4c0b50, 0x0, 0xb, 0x4301ad4c0e30, 0x4301ad0be3a0
0x439144c1bd70:[0x41800039dde7]AcpiExResolveToValue@vmkernel#nover+0x67 stack: 0x4301ad4c1e10, 0x4301ad0be3a0, 0x4301ad4c0b50, 0x4301ad2ea830, 0x11
0x439144c1bdb0:[0x418000393a60]AcpiDsEvaluateNamePath@vmkernel#nover+0x94 stack: 0x4301ad4c0b50, 0x0, 0x4301ad2ea830, 0x418000394163, 0x4301ad42ae80
0x439144c1bde0:[0x418000394163]AcpiDsExecEndOp@vmkernel#nover+0x39b stack: 0x4301ad4c0b50, 0x4301ad4c0b50, 0x0, 0x410016626855, 0x4301ad4c0b80
0x439144c1be20:[0x4180003a5f82]AcpiPsParseLoop@vmkernel#nover+0xd2 stack: 0x4301ad2ea830, 0x4301ad2ea830, 0x0, 0x0, 0x430100000003
0x439144c1be90:[0x4180003a6eb9]AcpiPsParseAml@vmkernel#nover+0x219 stack: 0x4301ad4c0f80, 0x4301ad464468, 0x0, 0x4180003a75cd, 0x430600000003
0x439144c1bec0:[0x4180003a75cd]AcpiPsExecuteMethod@vmkernel#nover+0x125 stack: 0x4301ad464450, 0x4301ad464470, 0x4301ad0f5f00, 0x0, 0x430614ff5640
0x439144c1bf00:[0x4180003a17b5]AcpiNsEvaluate@vmkernel#nover+0x15d stack: 0x4301ad4bdcf0, 0x4301ad464450, 0x430614ff57c0, 0x418000395ff2, 0x430614ff5080
0x439144c1bf30:[0x418000395ff2]AcpiEvAsynchExecuteGpeMethod@vmkernel#nover+0x112 stack: 0x430614ff57c0, 0x4180000ca3ed, 0x4307ad8b3050, 0x0, 0x4301c0b31050
0x439144c1bf50:[0x4180000ca3ed]helpFunc@vmkernel#nover+0x3c5
0x439144c1bfe0:[0x4180002cb675]CpuSched_StartWorld@vmkernel#nover+0x99 stack: 0x0, 0x0, 0x0, 0x0, 0x0
Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Environment
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
Cause
Page fault occurred to access a particular address causing the CPU bitflip.
Resolution
Intel has recently fixed some errata with BIOS and microcode updates. The Intel Microcode updates has been released as part of ESXi patch.
Upgrade the ESXi host to VMware ESXi 6.7, Patch Release ESXi670-201911001,VMware ESXi 6.5, Patch Release ESXi650-201911001 and VMware ESXi 6.0, Patch Release ESXi600-201911001.
Alternatively contact your hardware vendor to download the latest BIOS which contains the microcode version shown in the release notes for the above patches and also diagnose hardware.
Workaround:
No known workaround.
Upgrade the ESXi host to VMware ESXi 6.7, Patch Release ESXi670-201911001,VMware ESXi 6.5, Patch Release ESXi650-201911001 and VMware ESXi 6.0, Patch Release ESXi600-201911001.
Alternatively contact your hardware vendor to download the latest BIOS which contains the microcode version shown in the release notes for the above patches and also diagnose hardware.
Workaround:
No known workaround.
Additional Information
For more information also refer to VMware KB : Enabling Hypervisor-Specific Mitigations for Machine Check Error on Page Size Change (MCEPSC) Speculative-Execution vulnerability (CVE-2018-12207)
Impact/Risks:
Requires ESXi reboot.
Impact/Risks:
Requires ESXi reboot.
Feedback
Yes
No
Powered by
