Cannot delete a vSAN disk group or remove the disk after detecting it as faulty

Products

• VMware vSAN

Issue/Introduction

Symptoms

• After a disk failure, the disk/disk group cannot be deleted.

• Error message while deleting disk group:
  General vSAN error. vSAN disk data evacuation resource check has failed for disk or disk-group naa.58cxxxxxx (527c3109-ce7a-xxx-xxxx-xxxxxxxx) with mode noAction on host hostname.com. Go to vSAN Data Migration Pre-Check page for more details.

• Error message while deleting disk:
  A general system error occurred: Falled to decommission disk naa.50xxxxxxxxx: Failed to update diskState metadata for disk naa.50xxxxxxxxx with exception: Failed to write partition.

• The disk group may show as mounted or unmounted in the UI. It may also throw the below warning:
  Disk(s) 52a5e6e0-5359-xxxx-xxx-xxxxxxx , 52299652-2a60-xxxx-xxx-xxxxxxx , 52108e7e-7caa-xxxx-xxx-xxxxxxx , 52a8efb7-3e54-xxxx-xxx-xxxxxxx , 527eee1c-f821-xxxx-xxx-xxxxxxx are unmounted, but are part of a mounted disk group.

• Manually removing the disk group may fail with the below errors:
  Unable to remove device: Failed to get VsanInfo operation lock for diskOpLock, an operation is currently in progress(locked pid: 0), error: /tmp/.vsanDiskOpLock.lock.LOCK: timeout waiting for lock after 30 seconds. Lock is currently held by process 2167523 (vsanesxcmd: /usr/lib/vmware/vsan/bin/vsanesxcmd storage diskgroup mount -s naa.58cexxxxxxx)
Unable to remove device: Disk naa.58cexxxxxxx is not writable: Failed to write partition

• Trying to read the partition may fail with the below error:
  Error: Could not stat device /vmfs/devices/disks//vmfs/devices/disks/naa.58cexxxxxxx - No such file or directory.
Unable to get device /vmfs/devices/disks//vmfs/devices/disks/naa.58cexxxxxxx

Issue Validation

• Log entries similar to below may be seen in /var/run/log/vmkernel.log:
  YYYY-MM-DDTHH:MM:SSZ cpu4:2098069) ScsiDeviceIO: 4176: Cmd (0x45b9fc931fc6) 0x65, CmdSN Ox1352 from world 2102614 to dev "naa.50xxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0x0

• This sense code 0x7 0x27 means:

  Sense Key	[0x7]	DATA PROTECT
  Additional Sense Data	27/00	WRITE PROTECTED

  Meaning vSAN is unable to write to the device.

Environment

VMware vSAN

Cause

The disk has completely gone offline and ESXi host is not able to communicate with it.

Resolution

• In these conditions, unplug or remove the faulty disk physically from the server.

• Once the disk is removed, follow steps in the KB How to manually remove and recreate a vSAN disk group using esxcli to delete the disk group manually.

• Add the new disk to the physical server.

• Recreate the disk group on vSAN with the new disk.

Additional Information

How to remove a disk from a vSAN disk group/host