Virtual Machine in Hung State with 0B Storage Usage in vCenter Server Due to Unmapped Address Errors on Shared Datastore

Products

• VMware vSphere ESXi

Issue/Introduction

 

• VM was in hung state, storage/usage was showing 0B in vCenter UI.
  
  Log File: /var/log/vmkernel.log

[YYYY-MM-DDTHH:MM:SS] cpu37:2731967)WARNING: FS3J: 3297: Error committing txn callerID: 0xc1d0000f to slot 1: Address temporarily unmapped
[YYYY-MM-DDTHH:MM:SS] cpu37:2731967)Fil6: 4076: 'Data_Store_Name': Fil6 file IO (<FD c20 r25>) : Address temporarily unmapped
[YYYY-MM-DDTHH:MM:SS] cpu37:2731967)Fil6: 4042: ioCtx: 0x45ba6dd994c0, world: 2731967, overallStatus: Address temporarily unmapped.

• Storage device backing the datastore under "Datastore>Configure>Device Backing" may not be visible.

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

The logs show errors such as "Address temporarily unmapped," which suggests that the underlying storage might have temporarily become inaccessible, possibly due to datastore expand/extend operation. 

Lack of a rescan after changes results in the ESXi host not seeing the correct size, results in the VM's files being inaccessible at the time of the IO operation.

Resolution

• We recommend performing expand/extend datastores directly from vCenter rather than at the host client level to ensure that all hosts in the cluster are updated correctly.
• Ensure the rescan is performed across the entire cluster if the datastores are shared.
• Power cycle the affected virtual machine after ensuring the rescan process is complete to recover from the "hung" state.
• Engage storage vendor to further investigate the issue.

 

Additional Information

Please note not all Virtual machines running on the datastore which was recently extended/expanded will be affected and it depends on the IO operation VM was performing at that point in time.

Steps to RESCAN storage:

• Select the cluster where you have expanded the storage
• Select the ESXi Host -> Configure -> Storage -> Storage adapter -> Rescan storage
• Please select both the options: Scan for new Storage Devices & Scan for new VMFS Volumes
• Please ensure that you are performing this step on all the hosts in the cluster