ESXi host crashed with PSOD: #PF Exception 14 in world 2100600:storageRM IP 0x41801b1e588a addr 0x160
Article ID: 323597
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
cpu51:2100600)0x451b3d39aaa8:[0x41801b1e588a]UnmapCleanupInt@esx#nover+0x546 stack: 0x431301093b08, 0x418019ef7f35, 0x4313026cc9c0, 0x418000000010, 0x3bc2430e616d30c6
cpu51:2100600)0x451b3d39ab28:[0x41801b1e786a]Unmap_VolInit@esx#nover+0x1c7 stack: 0x4313026cc9c0, 0xda888555, 0x636f4c70616d6e75, 0x72656b616572426b, 0x431200000000
cpu51:2100600)0x451b3d39abc8:[0x41801b11bf74]Vol3OpenInt@esx#nover+0xca9 stack: 0x430f00000066, 0x430f0000009d, 0x18, 0x4180000000a0, 0x3732316334316535
cpu51:2100600)0x451b3d39aca8:[0x41801b11c4dd]Vol3GetVolumeObject@esx#nover+0x1f6 stack: 0x3463646165323331, 0x4309da2a49a0, 0x6d766c31343131, 0x0, 0x2e61616e00000000
cpu51:2100600)0x451b3d39aea8:[0x41801b11d6a0]Vol3GetObjectInt@esx#nover+0x165 stack: 0x451b3d39b07c, 0x418019f159ba, 0x1e8, 0x417fda211030, 0x430a0a25a830
cpu51:2100600)0x451b3d39af28:[0x418019e4bf50]FSSVec_GetObjectFS@vmkernel#nover+0x4d stack: 0x430a07ef8ec0, 0x451b3d39b07c, 0x430a0a25a830, 0x430a07ef8cd0, 0xf532
cpu51:2100600)0x451b3d39af68:[0x418019e498ef]FSS_GetObject@vmkernel#nover+0x44 stack: 0x451b3d39b07c, 0x451b3d39b070, 0x1, 0x418019e5d028, 0x431200000000
cpu51:2100600)0x451b3d39af98:[0x418019e5d027]OCReserveVolumeInt@vmkernel#nover+0x158 stack: 0x4312fe177070, 0x430a07ef8ec0, 0x3, 0x451b3d39b420, 0x14
cpu51:2100600)0x451b3d39afe8:[0x41801b140573]Fil3_GetObject@esx#nover+0x108 stack: 0x451b3d39b02e, 0x451b3d39bd68, 0x451b3d39b3e4, 0x5, 0x45a342ac0170
cpu51:2100600)0x451b3d39b148:[0x41801b11d57b]Vol3GetObjectInt@esx#nover+0x40 stack: 0x430e8dd638a0, 0x430a07ea3668, 0x430a07ea34f0, 0x451b3d39b41c, 0x10
cpu51:2100600)0x451b3d39b1c8:[0x418019e4bf50]FSSVec_GetObjectFS@vmkernel#nover+0x4d stack: 0x430a07ef8cd0, 0x451b3d39b41c, 0x10, 0x451b3d39b2f0, 0x1
cpu51:2100600)0x451b3d39b208:[0x418019e498ef]FSS_GetObject@vmkernel#nover+0x44 stack: 0x20, 0x451b3d39b2f0, 0x451b3d39b41c, 0x418019e5ccd9, 0xc180
cpu51:2100600)0x451b3d39b238:[0x418019e5ccd8]OCReserveInt@vmkernel#nover+0x191 stack: 0x0, 0x451b3d39b41c, 0x122400, 0xc180, 0x451b3d39b2f0
cpu51:2100600)0x451b3d39b278:[0x418019e5de04]OC_InsertObject@vmkernel#nover+0x149 stack: 0x451b3d39b41c, 0x430a07ef8cd0, 0x430a07c90678, 0x451b3d39bdb8, 0x451b3d39b41c
cpu51:2100600)0x451b3d39b2d8:[0x418019e4524c]FSSLookupInt@vmkernel#nover+0x26d stack: 0x430a07c90610, 0x430a07ea34f0, 0x0, 0x430a07c90678, 0x451b3d39bdb8
cpu51:2100600)0x451b3d39b338:[0x418019e452ff]FSS_Lookup@vmkernel#nover+0x24 stack: 0x0, 0x431608d3a010, 0x0, 0x41801a5972b7, 0x45a35e07f480
cpu51:2100600)0x451b3d39b368:[0x41801a5972b6]UserFileOpen@(user)#<None>+0x147 stack: 0x0, 0x80000003, 0x45a341bfa488, 0x451b3d39b578, 0xffffffffffffffff
cpu51:2100600)0x451b3d39b4e8:[0x41801a551999]UserObj_TraversePath@(user)#<None>+0x336 stack: 0x7bf, 0x451b3d39b596, 0x431608d48940, 0x431608d3a010, 0x60000000
cpu51:2100600)0x451b3d39bd88:[0x41801a59a467]LinuxFileDesc_Mkdir@(user)#<None>+0x90 stack: 0x0, 0x418019f1d398, 0x431608d48940, 0x1c3bd000005dc33, 0x3732316334316535
cpu51:2100600)0x451b3d39bee8:[0x41801a54b31b]User_LinuxSyscallHandler@(user)#<None>+0x180 stack: 0x451b3d39bfc8, 0x0, 0x0, 0x0, 0x0
cpu51:2100600)0x451b3d39bf28:[0x418019f2bd6c]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0x10b, 0x0, 0x0, 0x53, 0x1e93096147
cpu51:2100600)0x451b3d39bf38:[0x418019f63066]gate_entry@vmkernel#nover+0x67 stack: 0x0, 0x53, 0x1e93096147, 0x4d, 0x1e510a05f0
cpu51:2100600)ESC[45mESC[33;1mVMware ESXi 6.7.0 [Releasebuild-15160138 x86_64]ESC[0m
#PF Exception 14 in world 2100600:storageRM IP 0x41801b1e588a addr 0x160
- Reported PSOD backtrace:
cpu51:2100600)0x451b3d39aaa8:[0x41801b1e588a]UnmapCleanupInt@esx#nover+0x546 stack: 0x431301093b08, 0x418019ef7f35, 0x4313026cc9c0, 0x418000000010, 0x3bc2430e616d30c6
cpu51:2100600)0x451b3d39ab28:[0x41801b1e786a]Unmap_VolInit@esx#nover+0x1c7 stack: 0x4313026cc9c0, 0xda888555, 0x636f4c70616d6e75, 0x72656b616572426b, 0x431200000000
cpu51:2100600)0x451b3d39abc8:[0x41801b11bf74]Vol3OpenInt@esx#nover+0xca9 stack: 0x430f00000066, 0x430f0000009d, 0x18, 0x4180000000a0, 0x3732316334316535
cpu51:2100600)0x451b3d39aca8:[0x41801b11c4dd]Vol3GetVolumeObject@esx#nover+0x1f6 stack: 0x3463646165323331, 0x4309da2a49a0, 0x6d766c31343131, 0x0, 0x2e61616e00000000
cpu51:2100600)0x451b3d39aea8:[0x41801b11d6a0]Vol3GetObjectInt@esx#nover+0x165 stack: 0x451b3d39b07c, 0x418019f159ba, 0x1e8, 0x417fda211030, 0x430a0a25a830
cpu51:2100600)0x451b3d39af28:[0x418019e4bf50]FSSVec_GetObjectFS@vmkernel#nover+0x4d stack: 0x430a07ef8ec0, 0x451b3d39b07c, 0x430a0a25a830, 0x430a07ef8cd0, 0xf532
cpu51:2100600)0x451b3d39af68:[0x418019e498ef]FSS_GetObject@vmkernel#nover+0x44 stack: 0x451b3d39b07c, 0x451b3d39b070, 0x1, 0x418019e5d028, 0x431200000000
cpu51:2100600)0x451b3d39af98:[0x418019e5d027]OCReserveVolumeInt@vmkernel#nover+0x158 stack: 0x4312fe177070, 0x430a07ef8ec0, 0x3, 0x451b3d39b420, 0x14
cpu51:2100600)0x451b3d39afe8:[0x41801b140573]Fil3_GetObject@esx#nover+0x108 stack: 0x451b3d39b02e, 0x451b3d39bd68, 0x451b3d39b3e4, 0x5, 0x45a342ac0170
cpu51:2100600)0x451b3d39b148:[0x41801b11d57b]Vol3GetObjectInt@esx#nover+0x40 stack: 0x430e8dd638a0, 0x430a07ea3668, 0x430a07ea34f0, 0x451b3d39b41c, 0x10
cpu51:2100600)0x451b3d39b1c8:[0x418019e4bf50]FSSVec_GetObjectFS@vmkernel#nover+0x4d stack: 0x430a07ef8cd0, 0x451b3d39b41c, 0x10, 0x451b3d39b2f0, 0x1
cpu51:2100600)0x451b3d39b208:[0x418019e498ef]FSS_GetObject@vmkernel#nover+0x44 stack: 0x20, 0x451b3d39b2f0, 0x451b3d39b41c, 0x418019e5ccd9, 0xc180
cpu51:2100600)0x451b3d39b238:[0x418019e5ccd8]OCReserveInt@vmkernel#nover+0x191 stack: 0x0, 0x451b3d39b41c, 0x122400, 0xc180, 0x451b3d39b2f0
cpu51:2100600)0x451b3d39b278:[0x418019e5de04]OC_InsertObject@vmkernel#nover+0x149 stack: 0x451b3d39b41c, 0x430a07ef8cd0, 0x430a07c90678, 0x451b3d39bdb8, 0x451b3d39b41c
cpu51:2100600)0x451b3d39b2d8:[0x418019e4524c]FSSLookupInt@vmkernel#nover+0x26d stack: 0x430a07c90610, 0x430a07ea34f0, 0x0, 0x430a07c90678, 0x451b3d39bdb8
cpu51:2100600)0x451b3d39b338:[0x418019e452ff]FSS_Lookup@vmkernel#nover+0x24 stack: 0x0, 0x431608d3a010, 0x0, 0x41801a5972b7, 0x45a35e07f480
cpu51:2100600)0x451b3d39b368:[0x41801a5972b6]UserFileOpen@(user)#<None>+0x147 stack: 0x0, 0x80000003, 0x45a341bfa488, 0x451b3d39b578, 0xffffffffffffffff
cpu51:2100600)0x451b3d39b4e8:[0x41801a551999]UserObj_TraversePath@(user)#<None>+0x336 stack: 0x7bf, 0x451b3d39b596, 0x431608d48940, 0x431608d3a010, 0x60000000
cpu51:2100600)0x451b3d39bd88:[0x41801a59a467]LinuxFileDesc_Mkdir@(user)#<None>+0x90 stack: 0x0, 0x418019f1d398, 0x431608d48940, 0x1c3bd000005dc33, 0x3732316334316535
cpu51:2100600)0x451b3d39bee8:[0x41801a54b31b]User_LinuxSyscallHandler@(user)#<None>+0x180 stack: 0x451b3d39bfc8, 0x0, 0x0, 0x0, 0x0
cpu51:2100600)0x451b3d39bf28:[0x418019f2bd6c]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0x10b, 0x0, 0x0, 0x53, 0x1e93096147
cpu51:2100600)0x451b3d39bf38:[0x418019f63066]gate_entry@vmkernel#nover+0x67 stack: 0x0, 0x53, 0x1e93096147, 0x4d, 0x1e510a05f0
cpu51:2100600)ESC[45mESC[33;1mVMware ESXi 6.7.0 [Releasebuild-15160138 x86_64]ESC[0m
#PF Exception 14 in world 2100600:storageRM IP 0x41801b1e588a addr 0x160
- Relevant VMKernel Logs:
cpu51:2100600)WARNING: Heap: 3571: Heap vmfs3 already at its maximum size. Cannot expand.ESC[0m
cpu51:2100600)WARNING: Heap: 3571: Heap vmfs3 already at its maximum size. Cannot expand.ESC[0m
cpu51:2100600)WARNING: HBX: 303: No space for the journal on volume 5e14c127-da888555-0611-d094669d18a0 ("DATASTORE_NAME"). Volume will remain in read-only metadata mode with limited write support until journa$
cpu51:2100600)WARNING: Unmap6: 1002: No memory for unmap pool elementsESC[0m
pu51:2100600)WARNING: Unmap6: 2221: No memory for unmap cluster poolESC[0m
Environment
VMware vSphere ESXi 6.7
Cause
- PSOD/crash is hit on an error path when there is not enough memory space to create new heaps. Since the memory is under pressure all memory allocations are failing.
- Unmap_VolInit operation failed due to no memory condition and UnmapCleanupInt operation was failed to initialize.
Resolution
- VMware vSphere ESXi 6.7 P03 ESXi670-202008001
To download, go to Customer Connect Downloads .
Workaround:
- Reducing the memory load on an ESXi host should help here
- If issue persist, consider increasing total memory on ESXi host.
Feedback
Yes
No
Powered by
