Post datastore expansion, "Device Shrank" messages reported in VMkernel and volume is inaccessible.

Products

• VMware vSphere ESXi

Issue/Introduction

• Post datastore expansion via ESXi host client / CLI / vCenter UI, one or more hosts report "Device shrank" errors in VMkernel traces.
• VMFS volume is inaccessible.
• Inaccessible virtual machine.

Note: This is KB references a corner case, where the partition table entries were corrupted and correcting the partition table was required to fix the "Device Shrank" issue. Generally, host rescan or re-opening the VMFS volumes fixes the "Device Shrank" situations.

Environment

VMware vSphere 7.0.x

Cause

After a datastore is extended from host level, a storage rescan must be completed on all ESXi hosts accessing the datastore so that they detect the new size of the datastore.

If a storge rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded there (i.e. "stored size 17490245599 blocks") and its cached record of the original smaller device size (i.e. "actual size 17490243585 "). As the device appears to the host to be smaller than its stored size, a "Device Shrank" warning will be returned. 

ESXi host path /var/run/log/vmkernel.log :

2024-05-14T14:46:25.423Z cpu27:2100345 opID=5f2c3285)WARNING: LVM: 4107: [naa.600###############36:1] Device shrank (actual size 17490243585 blocks, stored size 17490245599 blocks)
2024-05-14T14:46:25.808Z cpu20:2395358)WARNING: NFS: 1226: Invalid volume UUID naa.600###############36:1
2024-05-14T14:46:25.824Z cpu20:2395358)FSS: 6092: No FS driver claimed device 'naa.600#################36:1': No filesystem on the device
2024-05-14T14:46:26.047Z cpu20:2100345 opID=5f2c3285)VC: 4616: Device rescan time 537 msec (total number of devices 128)
2024-05-14T14:46:26.047Z cpu20:2100345 opID=5f2c3285)VC: 4619: Filesystem probe time 361 msec (devices probed 122 of 128)
2024-05-14T14:46:26.047Z cpu20:2100345 opID=5f2c3285)VC: 4621: Refresh open volume time 4 msec

2024-05-14T14:47:32.844Z cpu28:2395502)WARNING: LVM: 4107: [naa.600#################36:1] Device shrank (actual size 17490243585 blocks, stored size 17490245599 blocks)
2024-05-14T14:47:32.989Z cpu17:2395366)WARNING: NFS: 1226: Invalid volume UUID naa.600#################36:1
2024-05-14T14:47:33.005Z cpu17:2395366)FSS: 6092: No FS driver claimed device 'naa.600#################36:1': No filesystem on the device
2024-05-14T14:47:33.413Z cpu28:2395502)VC: 4616: Device rescan time 539 msec (total number of devices 128)
2024-05-14T14:47:33.413Z cpu28:2395502)VC: 4619: Filesystem probe time 472 msec (devices probed 122 of 128)
2024-05-14T14:47:33.413Z cpu28:2395502)VC: 4621: Refresh open volume time 33 msec.

Resolution

If the above symptoms and issue matches, please contact Broadcom Support to investigate the issue.