ESXi hostd crashes due to redundant PVC snapshot chain in Cloud Native Storage

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi host management service (hostd) crashes repeatedly and auto-restarts due to memory exhaustion.

hostd.log: --> Panic: Memory exceeds hard limit. Panic

vobd.log: [esx.problem.hostd.core.dumped] /bin/hostd crashed (3 time(s) so far) and a core file may have been created at /var/core/hostd-zdump.002. This may have caused connections to the host to be dropped.

vsanvcmgmtd (vCenter Server):

info vsanvcmgmtd[14725] [vSAN@6876 sub=FcdService opId=75d7f338] Retrieve snapshot info begin: volumeId=1ff8b1ac-3daa-4c26-9e7b-##################
info vsanvcmgmtd[14725] [vSAN@6876 sub=vmomi.soapStub[5] opId=75d7f338] SOAP request returned HTTP failure; <<io_obj p:0x00007f8f714d7900, h:23, <TCP '##.##.##.## : 33650'>, <TCP '##.##.##.## : 1080'>>, /sdk>, method: retrieveSnapshotInfo; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Could not open or create change tracking file: api = DiskLib_BlockTrackGetEpoch, path->CValue() = /vmfs/volumes/vvol:176af88bf97c47d5-####################/naa.60002AC######################/_00e6/f06a9cf9afd14b#################-000001.vmdk"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f8f714d7900, h:23, <TCP '##.##.##.## : 33650'>, <TCP '##.##.##.## : 1080'>>, /sdk>]: retrieveSnapshotInfo
--> A general system error occurred: Could not open or create change tracking file: api = DiskLib_BlockTrackGetEpoch, path->CValue() = /vmfs/volumes/vvol:176af88bf97c47d5-##################/naa.60002AC0000000###############/_00e6/f06a9cf9afd##################-000001.vmdk"
--> }

Environment

ESXi 8.0

Cause

A Persistent Volume Claim (PVC) disk managed by Cloud Native Storage (CNS) developed a redundant or invalid snapshot chain, causing hostd memory exhaustion during parsing attempts.

Resolution

• Verify the impacted PVC is not actively mounted or in use by any running containers (e.g., OpenShift).

• Enable the ESXi Managed Object Browser (MOB) following Broadcom KB 340046.

• Access the deleteCatalogObject method via the host MOB at https://<REDACTED_HOSTNAME>/mob/?moid=ha-vstorage-object-manager&method=deleteCatalogObject&vmodl=1.

• Provide the following inputs to delete the catalog object: id: <id>1ff8b1ac-3daa-4c26-9e7b-###########</id> datastore: <datastore type="Datastore">vvol:176af88bf97c47d5-###############</datastore>

• Disable the ESXi MOB.