VMFS Datastore Inaccessibility Following Command-Line Resignature with Active Workloads
search cancel

VMFS Datastore Inaccessibility Following Command-Line Resignature with Active Workloads

book

Article ID: 434442

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

A VMFS datastore is reported as Inaccessible or Inconsistent across ESXi hosts,  Virtual Machines (VMs) remain "powered on" in memory (CPU/RAM) but lose all connectivity to their virtual disks (.vmdk).

You see the following log messages. 

Shell.log

2026-03-18T03:14:00.780Z In(14) shell[8554132]: [root]: esxcli storage vmfs snapshot resignature -u 5dcd985b-955a0e57-1ebd-############

2026-03-18T03:14:12.785Z In(14) shell[8554132]: [root]: esxcli storage vmfs snapshot resignature -u 5e2b0d5c-ded7ee55-7a16-############

2026-03-18T03:15:04.253Z In(14) shell[8554132]: [root]: esxcli storage vmfs snapshot resignature -u 602eebb1-48fd1393-b3f3-############


Hostd.log

2026-03-18T03:17:35.387Z Db(167) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] Got DSSYS change: [N11HostdCommon18DatastoreSystemMsgE:0x0000006c0305a8b0]DatastoreSystemMsg::DsChange{Type=DELETE, Msg=DatastoreDeletedMsg{DatastoreMsg{Type=1, MoId=5dcd985b-955a0e57-1ebd-############, Path=/vmfs/volumes/5dcd985b-955a0e57-1ebd-############}}}; DatastoreSystemMsg::DsChange{Type=DELETE, Msg=DatastoreDeletedMsg{DatastoreMsg{Type=1, MoId=5e2b0d5c-ded7ee55-7a16-############, Path=/vmfs/volumes/5e2b0d5c-ded7ee55-7a16-############}}}; DatastoreSystemMsg::DsChange{Type=DELETE, Msg=DatastoreDeletedMsg{DatastoreMsg{Type=1, MoId=602eebb1-48fd1393-b3f3-############, Path=/vmfs/volumes/602eebb1-48fd1393-b3f3-############}}}; DatastoreSystemMsg::DsChange{Type=ADD, Msg=DatastoreAddedMsg{DatastoreMsg{Type=1, MoId=69ba1885-156e6cc3-88d7-############, Path=/vmfs/volumes/69ba1885-156e6cc3-88d7-############}, Name=snap-volume###########1, WorkloadParams=WorkloadParams{outstandingIO(K1)=2, ioSize(K2)=20, readPercent(K3)=0.05, randomPercent(K4)=0.4}}}; DatastoreSystemMsg::DsChange{Type=ADD, Msg=DatastoreAddedMsg{DatastoreMsg{Type=1, MoId=69ba18b8-3bda62bd-ddeb-############, Path=/vmfs/volumes/69ba18b8-3bda62bd-ddeb-############}, Name=snap-volume###########2, WorkloadParams=WorkloadParams{outstandingIO(K1)=2, ioSize(K2)=20, readPercent(K3)=0.05, randomPercent(K4)=0.4}}};

2026-03-18T03:17:35.387Z Wa(164) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] UpdateStorageAccessibilityStatusInt: The cached datastore, 5dcd985b-955a0e57-1ebd-############ is not in the datastore inventory

2026-03-18T03:17:35.387Z In(166) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to false

2026-03-18T03:17:35.387Z In(166) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] VM config backing gone - try to mark VM invalid.

2026-03-18T03:17:35.387Z Db(167) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] Post act MarkBadLoad requested, current None

2026-03-18T03:17:35.387Z In(166) Hostd[2099132]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] Post act set from None to MarkBadLoad

2026-03-18T03:18:24.622Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx": Bad address.

2026-03-18T03:18:24.622Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx

2026-03-18T03:18:24.622Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx": Bad address.

2026-03-18T03:18:24.623Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx

2026-03-18T03:18:24.623Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] DictionaryLoad: Cannot open file "/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx": Bad address.

2026-03-18T03:18:24.625Z In(166) Hostd[2099158]: [Originator@6876 sub=Libs] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx

2026-03-18T03:18:24.625Z Db(167) Hostd[2099158]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] Generate policy from cfg message: Unable to load configuration file '/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx'.

2026-03-18T03:18:24.625Z Er(163) Hostd[2099158]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] Could not perform config check (storage not accessible): Fault cause: vim.fault.GenericVmConfigFault

2026-03-18T03:18:24.625Z In(166) Hostd[2099158]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5dcd985b-955a0e57-1ebd-############/VM1-######/VM1-######.vmx] VM state has changed; config check to be retried.

 

Environment

Vmware ESXi (All versions)

Cause

This issue is caused by a Metadata Identity Mismatch triggered when a VMFS volume is resignatured while hosting active file handles.

A VMFS resignature is a metadata change operation that changes the Volume’s Unique Identifier (UUID). When this is performed via the command line (esxcli storage vmfs snapshot resignature) while virtual machines are powered on, the following architectural conflict occurs:

  1. UUID Invalidation: The host updates the on-disk metadata with a new UUID, however this change is only reflected on the host that performed the resignature and it's not propagated to the other hosts in the cluster until a  storage rescan is performed on the cluster.

  2. Path Disruption: The original file paths (e.g., /vmfs/volumes/<OLD_UUID>/) used by the running hypervisor processes are deleted from the host's namespace.

  3. Handle Loss: Since ESXi cannot "live-remap" an active process to a new UUID, the running VMs lose their connection to persistent storage. They continue to run in memory because the process hasn't been killed, but any disk I/O request fails because the backing path no longer exists.

Resolution

1. Force Termination of "In-Memory" VMs via vCenter

Because the VMs have lost their backing storage, a "Graceful Shutdown" via the Guest OS is impossible. You must perform a Hard Stop to release the stale file handles and metadata heartbeats.

  1. Log in to the vSphere Client.

  2. Locate the impacted Virtual Machine in the inventory.

  3. Right-click the VM and select  Power > Shut Down Guest OS

    • Note: If the standard "Shutdown guest OS" task fails or hangs due to the inaccessible datastore, proceed to the next step.

  4. If the VM remains "Running" in the UI but is non-responsive:

    • Right-click the VM and select Power > Power Off (this may fail).

      • Right-click the VM and select Power > Force stop (this may fail).

    • If unsuccessful, you must use the ESXi Command Line to kill the "zombie" process:

      • SSH to the host running the VM.

      • Run: esxcli vm process list to find the World ID.

      • Run: esxcli vm process kill --type=force --world-id <WID>.

2. Cleanup and Re-registration

  1. Unregister VMs: Once the processes are stopped, right-click the "Invalid" or "Inaccessible" VMs in vCenter and select Remove from Inventory.

  2. Synchronize Storage: Perform a cluster-wide storage rescan so all hosts recognize the new UUID:

    • Right-click the Cluster > Storage > Rescan Storage.

  3. Register VMs: Browse the newly resignatured datastore, right-click the .vmx file, and select Register VM.

  4. Power On: Start the VMs to establish fresh, valid file handles against the new UUID.

Powered by Wolken Software