VMFS datastore expansion fails
Article ID: 401385
Updated On:
Products
Issue/Introduction
An attempt to expand a VMFS datastore fails with the following message:Failed to expand VMFS datastore Datastore_Name - Cannot change the host configuration
Environment
ESXi 7.0.x
ESXi 8.0.x
VMFS 6
VMFS 5
Cause
Metadata corruption was observed in the volume.
The following is logged in the hostd log at the time of datastore expansion failure####-##-##T##:##Z Db(167) Hostd[2099146]: [Originator@6876 sub=Solo.Vmomi opID=mb0tlxll-2317781-auto-1doew-h5:########-##-##-#### sid=######## user=vpxuser:Domain_Name\User_ID] Arg datastore:####-##-##T##:##Z Db(167) Hostd[2098674]: --> 'vim.Datastore:########-########-####-############'####-##-##T##:##Z Db(167) Hostd[2099146]: [Originator@6876 sub=Solo.Vmomi opID=mb0tlxll-2317781-auto-1doew-h5:########-##-##-#### sid=######## user=vpxuser:Domain_Name\User_ID] Arg spec:####-##-##T##:##Z Db(167) Hostd[2098674]: --> (vim.host.VmfsDatastoreExpandSpec) {####-##-##T##:##Z Db(167) Hostd[2098674]: --> diskUuid = "0200040000################################506f77657253",####-##-##T##:##Z Db(167) Hostd[2098674]: --> partition = (vim.host.DiskPartitionInfo.Specification) {####-##-##T##:##Z Db(167) Hostd[2098674]: --> partitionFormat = "gpt",####-##-##T##:##Z Db(167) Hostd[2098674]: --> chs = (vim.host.DiskDimensions.Chs) {####-##-##T##:##Z Db(167) Hostd[2098674]: --> cylinder = 1737770,####-##-##T##:##Z Db(167) Hostd[2098674]: --> head = 255,####-##-##T##:##Z Db(167) Hostd[2098674]: --> sector = 63####-##-##T##:##Z Db(167) Hostd[2098674]: --> },####-##-##T##:##Z Db(167) Hostd[2098674]: --> totalSectors = 27917287424,####-##-##T##:##Z Db(167) Hostd[2098674]: --> partition = (vim.host.DiskPartitionInfo.Partition) [####-##-##T##:##Z Db(167) Hostd[2098674]: --> (vim.host.DiskPartitionInfo.Partition) {####-##-##T##:##Z Db(167) Hostd[2098674]: --> partition = 1,####-##-##T##:##Z Db(167) Hostd[2098674]: --> startSector = 2048,####-##-##T##:##Z Db(167) Hostd[2098674]: --> endSector = 27917287390,####-##-##T##:##Z Db(167) Hostd[2098674]: --> type = "0xfb",####-##-##T##:##Z Db(167) Hostd[2098674]: --> guid = "AA31E02A400F11DB9590000C2911D1B8",####-##-##T##:##Z Db(167) Hostd[2098674]: --> logical = false,####-##-##T##:##Z Db(167) Hostd[2098674]: --> attributes = 0,####-##-##T##:##Z Db(167) Hostd[2098674]: --> partitionAlignment = 1048576####-##-##T##:##Z Db(167) Hostd[2098674]: --> }####-##-##T##:##Z Db(167) Hostd[2098674]: --> ],####-##-##T##:##Z Db(167) Hostd[2098674]: --> },####-##-##T##:##Z Db(167) Hostd[2098674]: --> extent = (vim.host.ScsiDisk.Partition) {####-##-##T##:##Z Db(167) Hostd[2098674]: --> diskName = "naa.################################",####-##-##T##:##Z Db(167) Hostd[2098674]: --> partition = 1####-##-##T##:##Z Db(167) Hostd[2098674]: --> }####-##-##T##:##Z Db(167) Hostd[2098674]: --> }####-##-##T##:##Z In(166) Hostd[2099146]: [Originator@6876 sub=Solo.Vmomi opID=mb0tlxll-2317781-auto-1doew-h5:########-##-##-#### sid=######## user=vpxuser:Domain_Name\User_ID] Throw vim.fault.HostConfigFault
vmkernel log####-##-##T##:##Z Wa(180) vmkwarning: cpu12:2099161)WARNING: Res3: 7661: Volume ########-########-####-############ ("Datastore_Name") might be damaged on the disk. Resource cluster metadata corruption has been detected.####-##-##T##:##Z Wa(180) vmkwarning: cpu12:2099161)WARNING: FS3: 636: VMFS volume Datastore_Name/########-########-####-############ on naa.################################:1 has been detected corrupted####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 639: While filing a PR, please report the names of all hosts that attach to this LUN, tests that were running on them,####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 662: and upload the dump by `voma -m vmfs -f dump -d /vmfs/devices/disks/naa.################################:1 -D X`####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 665: where X is the dump file name on a DIFFERENT volume####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 375: FS3RCMetaVMFS6 0 0 0 0 0 0 0 0 0####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 381: 0 0 0 0 0 0 0 0 0 0####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 386: 0 0 00000000-00000000-0000-000000000000####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 391: 0 0 0 0 0 0 0 0####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 400: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 402: 0 0####-##-##T##:##Z In(182) vmkernel: cpu12:2099161)FS3: 410: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0####-##-##T##:##Z Wa(180) vmkwarning: cpu12:2099161)WARNING: [type 1] Invalid totalResources 0 (cluster 0).####-##-##T##:##Z Wa(180) vmkwarning: cpu12:2099161)WARNING: [type 1] Invalid totalResources 0 (cluster 0).
vobd log####-##-##T##:##Z In(14) vobd[2097573]: [vmfsCorrelator] 615322814538us: [vob.vmfs.resource.corruptondisk] Volume ########-########-####-############ ("Datastore_Name") might be damaged on the disk. Resource cluster metadata corruption has been detected.####-##-##T##:##Z In(14) vobd[2097573]: Event rate limit reached. Dropping vprob: esx.problem.vmfs.resource.corruptondisk
Resolution
If the symptoms above align with the issue you are observing, please migrate all the VMs that you can out of the affected datastore and contact Broadcom Support for assistance in attempting a VOMA fix to try to recover the remaining VMs.
Feedback
