Post datastore expansion, "Device Shrank" messages reported in VMkernel and volume is inaccessible.

Products

• VMware vSphere ESXi

Issue/Introduction

• Post datastore expansion via ESXi host client / CLI / vCenter UI, one or more hosts report "Device shrank" errors in VMkernel traces.
• VMFS volume is inaccessible.
• Inaccessible virtual machine.

Note: This is KB references a corner case, where the partition table entries were corrupted and correcting the partition table was required to fix the "Device Shrank" issue. Generally, host rescan or re-opening the VMFS volumes fixes the "Device Shrank" situations.

Environment

VMware vSphere ESXi 7.0.x

Cause

After a datastore is extended from host level, a storage rescan must be completed on all ESXi hosts accessing the datastore so that they detect the new size of the datastore.

If a storge rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded there (i.e. "stored size 17490245599 blocks") and its cached record of the original smaller device size (i.e. "actual size 17490243585"). As the device appears to the host to be smaller than its stored size, a "Device Shrank" warning will be returned. 

e.g.) ESXi host path /var/run/log/vmkernel.log :

YYYY-MM-DDThh:mm:ss.fffZ cpu27:2100345 opID=########)WARNING: LVM: 4107: [naa.600###############36:1] Device shrank (actual size 17490243585 blocks, stored size 17490245599 blocks)
YYYY-MM-DDThh:mm:ss.fffZ cpu20:2395358)WARNING: NFS: 1226: Invalid volume UUID naa.600###############36:1
YYYY-MM-DDThh:mm:ss.fffZ cpu20:2395358)FSS: 6092: No FS driver claimed device 'naa.600#################36:1': No filesystem on the device
YYYY-MM-DDThh:mm:ss.fffZ cpu20:2100345 opID=########)VC: 4616: Device rescan time 537 msec (total number of devices 128)
YYYY-MM-DDThh:mm:ss.fffZ cpu20:2100345 opID=########)VC: 4619: Filesystem probe time 361 msec (devices probed 122 of 128)
YYYY-MM-DDThh:mm:ss.fffZ cpu20:2100345 opID=########)VC: 4621: Refresh open volume time 4 msec

YYYY-MM-DDThh:mm:ss.fffZ cpu28:2395502)WARNING: LVM: 4107: [naa.600#################36:1] Device shrank (actual size 17490243585 blocks, stored size 17490245599 blocks)
YYYY-MM-DDThh:mm:ss.fffZ cpu17:2395366)WARNING: NFS: 1226: Invalid volume UUID naa.600#################36:1
YYYY-MM-DDThh:mm:ss.fffZ cpu17:2395366)FSS: 6092: No FS driver claimed device 'naa.600#################36:1': No filesystem on the device
YYYY-MM-DDThh:mm:ss.fffZ cpu28:2395502)VC: 4616: Device rescan time 539 msec (total number of devices 128)
YYYY-MM-DDThh:mm:ss.fffZ cpu28:2395502)VC: 4619: Filesystem probe time 472 msec (devices probed 122 of 128)
YYYY-MM-DDThh:mm:ss.fffZ cpu28:2395502)VC: 4621: Refresh open volume time 33 msec.

Resolution

If the above symptoms and issue matches, please contact Broadcom Support to investigate the issue.

Additional Information

Japanese Version:
データストアの拡張後に VMkernel に "Device Shrank" というメッセージが出力されてボリュームにアクセスできない