After a datastore is expanded from the ESXi host client or CLI, one or more hosts report "Device shrank" and the datastore becomes inaccessible

Products

• VMware vSphere ESXi

Issue/Introduction

Symptom:

• A datastore is extended from the ESXi host client or the ESXi CLI. Shortly after, on one or more hosts, vmkernel.log reports "Device shrank": 
  
  YYYY-MM-DDTHH:MM:DDZ cpu28:2100198 opID=8480b361)World: 11943: VC opID 17b6faf3 maps to vmkernel opID 8480b361 
YYYY-MM-DDTHH:MM:DDZ cpu28:2100198 opID=8480b361)WARNING: LVM: 4087: [<naa device id>:1] Device shrank (actual size 6442448863 blocks, stored size 8589819905 blocks) 
YYYY-MM-DDTHH:MM:DDZ cpu28:2100198 opID=8480b361)LVM: 13544: Deleting device <naa device id:1>dev OpenCount: 1, postRescan: False 
YYYY-MM-DDTHH:MM:DDZ cpu23:2099366 opID=a52a94b2)World: 11943: VC opID 17b6fafd maps to vmkernel opID a52a94b2
  
  
• And VM I/O fails with "Address temporarily unmapped":
  
  YYYY-MM-DDTHH:MM:DDZ cpu43:2097598)WARNING: FS3J: 3179: Error committing txn callerID: 0xc1d0000f to slot 1: Address temporarily unmapped 
YYYY-MM-DDTHH:MM:DDZ cpu43:2097598)BC: 410: write to vmware.log (f532 28 3 5e1455f8 b1accc9b 2500281a c0121b5 48004c4 245d 0 0 0 0 0) 4356 bytes failed: Address temporarily unmapped
  
  
• The datastore and VMs accessing the datastore become inaccessible.

Environment

VMware vSphere 7.x

VMware vSphere 8.x

Cause

Note: Increasing the size of the datastore backing lun from the storage side doesn't automatically increase the size of the datastore on the VMware infra. To increase the size of the datastore -  Extending or increasing a datastore through vCenter Server fails

• After a datastore is extended from host level, a storage rescan must be completed on all ESXi hosts accessing the datastore so that they detect the new size of the datastore.
  
  
• If a storge rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded there (i.e. "stored size 8589819905 blocks") and its cached record of the original smaller device size (i.e. "actual size 6442448863"). As the device appears to the host to be smaller than its stored size, a "Device Shrank" warning will be returned. 
   
  
   
• If the datastore is mounted across multiple clusters and all the hosts have not been rescanned after the size of the lun has been increased then we see this issue since vCenter is displaying all the details in the UI after connecting with one of the host randomly.
  
• Subsequently, if a write is attempted to the extended part of the volume, from a host which has not been rescanned then it will trigger a warning "Address temporarily unmapped" because it has determined that the device shrank and that the addressed part of the device is unavailable.

Resolution

• A cluster or datacenter level rescan will trigger all hosts to detect the new size of the datastore and the datastore will become accessible. VMs which became inaccessible may need to be restarted.
   

Additional Information

• Impact/Risks:
  The datastore will be reported as inaccessible on hosts on which "Address temporarily unmapped" is reported. VMs will become inaccessible.

• When a datastore is expanded and a storage rescan is not run, or if it fails on a host, other errors/symptoms can be seen as well. For reference: Facing an I/O error on a VM which is hosted on VMware Environment.