ESXi Host Crashes with PSOD: DE Exception 0 in VMK Swapper.
search cancel

ESXi Host Crashes with PSOD: DE Exception 0 in VMK Swapper.

book

Article ID: 402494

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

A VMFS datastore has been identified with metadata corruption. Until the corruption is addressed, any ESXi host with the corrupted datastore mounted is at risk of experiencing instability, including potential PSODs.

During the PSOD event, a backtrace similar to the one shown below may be recorded on the impacted host:

0x45397679b730:[0x4200214f6a2c]Res3IsValidPtrBlockAddr64VMFS6@esx#nover+0x4c stack: 0x43147cf51800, 0x45397679b830, 0x43147d46be20, 0x4200214f7814, 0x0
0x45397679b760:[0x4200214f7813]Res3ResolveIntVMFS6@esx#nover+0x3c stack: 0x45b9ec72b500, 0x45b9ec72b700, 0x0, 0x45d9c7f869c0, 0x1
0x45397679b7c0:[0x42002148c2bc]Fil6_ResolveFileOffsetVector@esx#nover+0xcf1 stack: 0x4a, 0x0, 0x0, 0x0, 0x0
0x45397679b8d0:[0x420021491bea]Fil6_FileIOInt@esx#nover+0xcb3 stack: 0x45d9805b2f00, 0x43147cffa1e0, 0x43147fc9d260, 0x430b98602c00, 0x45397679bc5c
0x45397679ba30:[0x42002149396b]Fil6_FileIOIntWithRetry@esx#nover+0x2a4 stack: 0x0, 0x0, 0xa0014000a0014, 0x32000a, 0x0
0x45397679bae0:[0x420021493a93]Fil6_FileIO@esx#nover+0x104 stack: 0x45d9efc00f20, 0x45d9805b2f00, 0x80011380000000, 0x420020184da2, 0x4301ff4014c0
0x45397679bbd0:[0x420020042038]FSSVec_FileIO@vmkernel#nover+0x21 stack: 0x20943c8f, 0x4200206c2608, 0x45397679bc5c, 0x38, 0xcd618290
0x45397679bbf0:[0x4200206c2607]FSSFileIO@vmkernel#nover+0x17c stack: 0xcd618290, 0x430b98d6e000, 0x38, 0x430b98601b40, 0x45d9809c7f58
0x45397679bc50:[0x4200206c27d1]FSS_AsyncFileIO@vmkernel#nover+0xe stack: 0x45399169f000, 0x420020200ad4, 0x0, 0x4000000005, 0x4315cd61cd40
0x45397679bc70:[0x420020200ad3]SwapIOClusterWriteWrapper@vmkernel#nover+0x620 stack: 0x4315cd61cd40, 0x430b98602f10, 0x20, 0x4315cd618290, 0xcd60ec60
0x45397679bdc0:[0x420020200fd4]SwapIO_SwapOutInt@vmkernel#nover+0x95 stack: 0x4315cd618290, 0x3, 0x4315cd618290, 0x45399169f000, 0x4315cd61dc30
0x45397679be10:[0x4200201f6dae]SwapVMKSwapOrRetry@vmkernel#nover+0x6e3 stack: 0x200, 0x3, 0x58, 0xc42960468a8c8, 0x45397679bec0
0x45397679bf60:[0x42002015ba1c]HelperQueueFunc@vmkernel#nover+0x19d stack: 0x430b98601238, 0x45397679f000, 0x0, 0x0, 0x0
0x45397679bfe0:[0x4200206dc88e]CpuSched_StartWorld@vmkernel#nover+0xbf stack: 0x0, 0x420020144fb0, 0x0, 0x0, 0x0

While checking /var/run/log/vmkernel.log similar entries may be observed:

WARNING: Fil3: 7386: Invalid Address <5, 1> in FD <##,###> on volume 'Corrupt_Datastore_Name' ESC[0m
WARNING: Fil3: 7386: Invalid Address <5, 1> in FD <##,###> on volume 'Corrupt_Datastore_Name' ESC[0m
FS3DM: 2375: status Invalid metadata copying ### extents between two files, bytesTransferred = 0 extentsTransferred: 0
WARNING: Fil3: 8362: Data Move error: Invalid metadataESC[0m
WARNING: SVM: 2891: scsi###:### Failed SVMFDSIoctlMoveData: TimeoutESC[0m
SVM: 2944: scsi###:### Completed copy in 0 ms. vmmLeaderID = #########.
WARNING: FSR: 1271: ################## S: Swap copy failed: TimeoutESC[0m
Vol3: 4768: Failed to get object 28 type 1 uuid #######-#########-######-############ FD 0 gen 0 :Invalid metadata
Vol3: 4768: Failed to get object 28 type 2 uuid #######-#########-######-############ FD 4 gen 1 :Invalid metadata

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

VMFS metadata corruption can occur due to several reasons, including:

  • Unplanned power outages or improper shutdowns

  • Storage hardware failures (e.g., disk or controller issues)

  • Misconfigured multipathing or zoning

  • Firmware or driver incompatibility

  • Concurrent or improper access to shared datastores

  • Software bugs or kernel panics during I/O operations

These factors can lead to inconsistencies in the filesystem structure, making the datastore unstable and potentially causing host crashes (PSOD) if not addressed promptly.

Resolution

To mitigate the risk and begin remediation, perform the following actions:

  1. Un-mount the Affected Datastore:

    • Remove or un-mount the corrupted datastore from all ESXi hosts.

    • This step is critical to prevent further crashes or data inconsistencies.

  2. Engage Broadcom Support:

    • Contact Broadcom (VMware) Support to perform further investigation using the VOMA (VMware Ondisk Metadata Analyzer) utility.

    • VOMA is used to scan and repair on-disk metadata on VMFS datastores and must be run under guided support conditions.

    • Broadcom Support can also help confirm the exact device or datastore that is corrupted and requires attention.

Additional Information

Please do not attempt to run the VOMA utility independently in a production environment without guidance from Broadcom Support, as improper use may lead to data loss.

When contacting support, reference this KB article to provide context and expedite resolution.

 

Powered by Wolken Software