Unable to delete local VMFS datatore created on boot lun
Article ID: 371045
Updated On:
Products
Issue/Introduction
Symptoms:
- Unable to delete Local Datastore
- Deleting 'Datastore_name' from host GUI and vCenter GUI reports 'An error occurred during host configuration'.
- While trying to delete a partition on a 4KN SWE disk the operation fails with the error message "Read-only file system during write on /dev/disks/<device name>" and the vmkernel.log has the entry "in-use partition <part num>, modification is not supported".
- Local VMFS datastore automatically created on the ESXi boot lun cannot be removed/deleted. Deletion fails from both vSphere Client and Host Client, as well as from the ESXi command line.
#esxcli storage core device capacity listDevice Physical Blocksize Logical Blocksize Logical Block Count Size Format Type------------------------------------ ------------------ ----------------- ------------------- ---------- -----------naa.6f4e########################23c0 4096 512 936640512 457344 MiB4Kn SWE
#esxcli storage core device list -d naa.6f4e########################23c0naa.6f4e########################23c0 Display Name: Local DELL Disk (naa.6f4e########################23c0) Has Settable Display Name: true Size: 457344 Device Type: Direct-Access Multipath Plugin: HPP Devfs Path: /vmfs/devices/disks/naa.6f4e########################23c0 Vendor: DELL Model: RAID Revision: 8-29 Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: unknown Attached Filters: VAAI Status: unsupported Other UIDs: vml.02000000006f4e########################23c0524149442020 Is Boot Device: true
#partedUtil getptbl /vmfs/devices/disks/naa.6f4e########################23c0
Device: /vmfs/devices/disks/naa.6f4e########################23c0Partition table:gpt7287 255 63 9366405121 64 204863 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 1285 208896 8595455 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 06 8597504 16984063 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 07 16986112 268435455 4EB2EA3978554790A79EFAE495E21F8D vmfsl 08 268437504 936640471 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
Usable sectors:48 936640471
#partedUtil delete /vmfs/devices/disks/naa.6f4e########################23c0 8Error: Read-only file system during write on /dev/disks/naa.6f4e########################23c0 Unable to delete partition 8 from device /vmfs/devices/disks/naa.6f4e########################23c0
vmkernel.log:vmkwarning: cpu73:2100332)WARNING: Partition: 2124: naa.6f4e########################23c0: in-use partition 7 modification is not supported
hostd.log: Hostd[2099014]: [Originator@6876 sub=Partitionsvc opID=esxhost1 sid=xxxx user=root] Unable to setptbl on /vmfs/devices/disks/naa.6f4e########################23c0 Hostd[2098996]: --> Status: 255 Hostd[2098996]: --> Output: gpt Hostd[2098996]: --> 0 0 0 0 Hostd[2098996]: --> 1 64 204863 C12A7328F81F11D2BA4B00A0C93EC93B 128 Hostd[2098996]: --> 5 208896 8595455 EBD0A0A2B9E5443387C068B6B72699C7 0 Hostd[2098996]: --> 6 8597504 16984063 EBD0A0A2B9E5443387C068B6B72699C7 0 Hostd[2098996]: --> 7 16986112 268435455 4EB2EA3978554790A79EFAE495E21F8D 0 Hostd[2098996]: --> Hostd[2098996]: --> Error: Error: Read-only file system during write on /dev/disks/naa.6f4e########################23c0 Hostd[2098996]: --> SetPtableGpt: Unable to commit to disk Hostd[2098996]: --> Hostd[2099031]: [Originator@6876 sub=Hostsvc opID=esxhost1 sid=xxxx user=root] DeletePartition: Failed to update partition information : Fault cause: vim.fault.HostConfigFault Hostd[2099031]: [Originator@6876 sub=Hostsvc.DatastoreSystem opID=esxhost1 sid=xxxx user=root] DestroyVmfsDatastore: can't delete partition 8 on lun naa.6f4e########################23c0 Hostd[2099031]: [Originator@6876 sub=Hostsvc.DatastoreSystem opID=esxhost1 sid=xxxx user=root] RemoveDatastore: Failed to remove backend for datastore localdatastore1. Hostd[2099031]: [Originator@6876 sub=Hostsvc.DatastoreSystem opID=esxhost1 sid=xxxx user=root] File descriptors in use:
Environment
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
Cause
Issue is seen when using 4K HDD Device of format 4Kn SWE for boot lun. On the physical device which is 4K the GPT entries start from the second sector and each sector holds 32 GPT entries (128 bytes each). The disk contains 8 partition, all of which are stored on the 2nd sector.
When this is translated to 512 block size, the GPT entries are calculated to start from the 16th sector and all the 8 partition entries are stored on the 16th partition. This is a mistake since for 512 block size each partition will only be able to hold 4 GPT partition entires. So the first 4 GPT entries will be stored on sector 16 and the next 4 GPT entries will be stored on sector 17.
During partition delete operation, PSA will recreate the partition table without the deleted partition entry and try to write the new partition table on the disk. In this operation PSA will check if any of the partitions which are in use are being modified and fail the operation if true.
To check if any of the partitions are modified it will compare the partition entries on disk to the new partition entries. But since the calculation mistake assumes that all the entries are on sector 16 the partition deletion code will read the first 4 partition entries (from sector 16) to compare with the last 4 partition entries and erroneously compare the partition 3 with partition 7. Since partition 7 is an OSDATA partition it will always have an handle open and PSA will assume that it is modifying an opened partition and fail the partition deletion operation.
Resolution
This issue is fixed in ESXi 8.0.3 P04 (ESXi 8.0 Update 3b).
Refer - VMware ESXi 8.0 Update 3b Release Notes
Feedback
