Unable to expand a VMFS 5/6 datastore for any volume of size that is greater than or equal to 16TB

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides the steps to successfully expand VMFS 5/6 datastore.

Symptoms:

• Unable to expand VMFS 5/6 datastore for any volume of size that is greater than  or equal to 16TB.
• ESXi Hosts Before ESXi 6.0 EP22, VMware ESXi 6.5 U2 and VMware ESXi 6.7 are subject to get affected with the issue.
• In the vCenter, the device shows the LUN is expanded but VMFS datastore expansion fails.
• Attempt to expand the VMFS datastore is successful. However, vCenter view will still show the original size.
• Running partedUtil command against the device shows the new expanded size. For more information, see Using the partedUtil command line utility on ESXi and ESX.
• Running the esxcli storage core device list command shows the size of the extended LUN.
• Running the vmkfstools -Ph /vmfs/volumes<datastore name> command shows the original size of datastore
• In the Hostd.log file, you see entries similar to:

2017-09-07T10:53:10.312Z info hostd[EB85B70] [Originator@6876 sub=Vimsvc.TaskManager opID=c48ad960 user=root] Task Created : haTask-ha-host-              vim.host.DatastoreSystem.expandVmfsDatastore-147483448
2017-09-07T10:53:10.312Z verbose hostd[EB85B70] [Originator@6876 sub=PropertyProvider opID=c48ad960 user=root] RecordOp ADD: recentTask["haTask-ha-host-vim.host.DatastoreSystem.expandVmfsDatastore-147483448"], ha-host. Sent notification immediately.
2017-09-07T10:53:10.312Z verbose hostd[EB85B70] [Originator@6876 sub=PropertyProvider opID=c48ad960 user=root] RecordOp ADD: recentTask["haTask-ha-host-vim.host.DatastoreSystem.expandVmfsDatastore-147483448"], ha-taskmgr. Applied change to temp map.
2017-09-07T10:53:10.312Z verbose hostd[9221870] [Originator@6876 sub=PropertyProvider opID=c48ad960 user=root] RecordOp ASSIGN: info, haTask-ha-host-vim.host.DatastoreSystem.expandVmfsDatastore-147483448. Applied change to temp map.
2017-09-07T10:53:10.313Z info hostd[9221870] [Originator@6876 sub=SysCommandPosix opID=c48ad960 user=root] ForkExec(/sbin/partedUtil) 211172
2017-09-07T10:53:10.531Z error hostd[9221870] [Originator@6876 sub=Hostsvc.FSVolumeProvider opID=c48ad960 user=root] VmkCtl GrowExtent failed: Unable to grow Filesystem, please see VMkernel log fordetails : Underlying device has no free space
2017-09-07T10:53:10.531Z warning hostd[9221870] [Originator@6876 sub=Hostsvc.DatastoreSystem opID=c48ad960 user=root] CheckAndUpdateVmfsDatastoreSpec: Error expanding extent #1 of volume /vmfs/volumes/58e72006-2c646e8e-1a44-90e2baba69fc, this extent is ignored.
2017-09-07T10:53:10.531Z info hostd[9221870] [Originator@6876 sub=Default opID=c48ad960 user=root] AdapterServer caught exception: vim.fault.HostConfigFault
2017-09-07T10:53:10.531Z info hostd[9221870] [Originator@6876 sub=Vimsvc.TaskManager opID=c48ad960 user=root] Task Completed : haTask-ha-host-vim.host.DatastoreSystem.expandVmfsDatastore-147483448 Status error
2017-09-07T10:53:10.531Z verbose hostd[9221870] [Originator@6876 sub=PropertyProvider opID=c48ad960 user=root] RecordOp ASSIGN: info, haTask-ha-host-vim.host.DatastoreSystem.expandVmfsDatastore-147483448. Applied change to temp map.
2017-09-07T10:53:10.531Z info hostd[9221870] [Originator@6876 sub=Solo.Vmomi opID=c48ad960 user=root] Activation [N5Vmomi10ActivationE:0x0e373ea8] : Invoke done [expandVmfsDatastore] on [vim.host.DatastoreSystem:ha-datastoresystem]
2017-09-07T10:53:10.531Z verbose hostd[9221870] [Originator@6876 sub=Solo.Vmomi opID=c48ad960 user=root] Arg datastore:
--> 'vim.Datastore:58e72006-2c646e8e-1a44-90e2baba69fc'
2017-09-07T10:53:10.531Z verbose hostd[9221870] [Originator@6876 sub=Solo.Vmomi opID=c48ad960 user=root] Arg spec:
--> (vim.host.VmfsDatastoreExpandSpec) {
--> diskUuid = "0200000000600507630080865d7800000000000018323134352020",
--> partition = (vim.host.DiskPartitionInfo.Specification) {
--> partitionFormat = "gpt",
--> chs = (vim.host.DiskDimensions.Chs) null,
--> totalSectors = <unset>,
--> partition = (vim.host.DiskPartitionInfo.Partition) [
--> (vim.host.DiskPartitionInfo.Partition) {
--> partition = 1,
--> startSector = 2048,
--> endSector = 92103188480,
--> type = "0xfb",
--> guid = "AA31E02A400F11DB9590000C2911D1B8",
--> logical = false,
--> attributes = 0,
--> partitionAlignment = <unset>
--> }
--> ]
--> },
--> extent = (vim.host.ScsiDisk.Partition) {
--> diskName = "naa.600507630080865d7800000000000018",
--> partition = 1
--> }
--> }
2017-09-07T10:53:10.531Z info hostd[9221870] [Originator@6876 sub=Solo.Vmomi opID=c48ad960 user=root] Throw vim.fault.HostConfigFault
2017-09-07T10:53:10.531Z info hostd[9221870] [Originator@6876 sub=Solo.Vmomi opID=c48ad960 user=root] Result:
--> (vim.fault.HostConfigFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""

•   In the vmkernel.log file, you see entries similar to:

          2017-09-07T11:20:46.855Z cpu30:212496)LVM: 10661: Using all available space (11972489773056).
     2017-09-07T11:20:46.862Z cpu30:212496)LVM: 10571: Successfully added space (0) on device naa.600507630080865d7800000000000018:1 to volume 58e72006-fc365a06-             0c72-90e2baba69fc
     2017-09-07T11:30:04.094Z cpu22:68831 opID=6a65f73a)LVM: 4014: [naa.600507630080865d7800000000000018:1] Device expanded (actual size 92341792769 blocks, stored           size 92103186433 blocks)

2017-09-07T11:30:04.111Z cpu22:68831 opID=6a65f73a)LVM: 4014: [naa.600507630080865d7800000000000018:1] Device expanded (actual size 92341792769 blocks, stored size 92103186433 blocks)

2017-09-07T11:30:04.117Z cpu22:68831 opID=6a65f73a)LVM: 14448: LVM device naa.600507630080865d7800000000000018:1 successfully expanded (new size:                47278997897728)<---------------

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

With VMFS 5 and VMFS 6, the Physical Extents (PE) grafting is failing when the volume of size greater than or equal to 16TB is extended.

Resolution

This issue is resolved in VMware ESXi 6.0 EP22, VMware ESXi 6.5 U2 and VMware ESXi 6.7, available at VMware Downloads.
For patch download go to the Customer Connect Patch Downloads page.

NOTE:- The datastores that are already impacted, we need to follow the workaround below and the patch is a fix to avoid any new occurrences of the issue being reported.

Workaround:

To work around this issue if you do not want to upgrade:

1. Storage vmotion the virtual machines on the affected datastore.
2. Format this LUN and present the LUN with the new size.
3. Create a VMFS 5/6 datastore. For more information, see the vSphere Storage Guide.

Note: Datastore expansion should be done after applying the fix where it has not failed or tried previously.

Additional Information

Impact/Risks:
Storage vmotion the VMs on the affected datastore.