After a datastore is expanded from the ESXi host client or CLI, one or more hosts report "Device shrank" and the datastore becomes inaccessible

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

A datastore is extended from the ESXi host client or the ESXi CLI. Shortly after, on one or more hosts, vmkernel.log reports "Device shrank": 

2020-04-02T09:01:23.252Z cpu28:2100198 opID=8480b361)World: 11943: VC opID 17b6faf3 maps to vmkernel opID 8480b361
2020-04-02T09:01:23.252Z cpu28:2100198 opID=8480b361)WARNING: LVM: 4087: [<naa device id>:1] Device shrank (actual size 6442448863 blocks, stored size 8589819905 blocks)
2020-04-02T09:01:23.252Z cpu28:2100198 opID=8480b361)LVM: 13544: Deleting device <naa device id:1>dev OpenCount: 1, postRescan: False
2020-04-02T09:01:25.336Z cpu23:2099366 opID=a52a94b2)World: 11943: VC opID 17b6fafd maps to vmkernel opID a52a94b2

And VM I/O fails with "Address temporarily unmapped":

2020-04-02T09:17:09.468Z cpu43:2097598)WARNING: FS3J: 3179: Error committing txn callerID: 0xc1d0000f to slot 1: Address temporarily unmapped
2020-04-02T09:17:09.468Z cpu43:2097598)BC: 410: write to vmware.log (f532 28 3 5e1455f8 b1accc9b 2500281a c0121b5 48004c4 245d 0 0 0 0 0) 4356 bytes failed: Address temporarily unmapped

The datastore and VMs accessing the datastore become inaccessible.

Environment

VMware vSphere 7.0.#

VMware vSphere 8.0.#

Cause

After a datastore is extended from host level, a storage rescan must be completed on all ESXi hosts accessing the datastore so that they detect the new size of the datastore.

If a storge rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded there (i.e. "stored size 8589819905 blocks") and its cached record of the original smaller device size (i.e. "actual size 6442448863"). As the device appears to the host to be smaller than its stored size, a "Device Shrank" warning will be returned. 
 
Subsequently, if a write is attempted to the extended part of the volume, the host will trigger a warning "Address temporarily unmapped" because it has determined that the device shrank and that the addressed part of the device is unavailable. 

Resolution

A cluster wide rescan will trigger all hosts to detect the new size of the datastore and the datastore will become accessible. 

VMs which became inaccessible may need to be restarted.

Additional Information

• Impact/Risks:
  The datastore will be reported as inaccessible on hosts on which "Address temporarily unmapped" is reported. VMs will become inaccessible.
  
  

 

• When a datastore is expanded and a storage rescan is not run, or if it fails on a host, other errors/symptoms can be seen as well. For reference: Facing an I/O error on a VM which is hosted on VMware Environment.