vSAN -- OSA - Deduplication & Compression enabled -- Identifying and Replacing a failed Cache or Capacity disk

Products

VMware vSAN 7.x VMware vSAN 8.x VMware vSAN 6.x

Issue/Introduction

When VMware vSAN is configured with Deduplication & Compression enabled, a failure of any single disk results in the failure of the entire Disk Group to which the disk belongs.

The associated vSAN Skyline Health test for Operation health reports the entire Disk Group as Offline.

Example Output:

This status is also reflected in Configure > vSAN > Disk Management.

Example Output:

In the example, one disk is marked as "Absent". Due to the nature of the failure, only the Disk UUID is displayed; the original disk name (e.g., naa.#####) is no longer visible

Environment

VMware vSAN 6.x, 7.x, 8.x (Original Storage Architecture - OSA) and Deduplication and Compression enabled

Cause

vSAN Deduplication occurs at the Disk Group level across the Cluster.

As a result, if a single disk in the Disk Group fails, the entire Disk Group fails.

The UI reflects the Disk Group failure, but it does not display the identifying information of the device that triggered the failure.

Resolution

To identify the specific device that caused the failure, follow these steps:

1.) Log in to the affected vSAN Host via SSH or KVM/physical console

2.) List vSAN disks via command

esxcli vsan storage list |less

The failed disk will show as "Unknown" and/or missing "Display Name"

Example Output:

Unknown:
Device: Unknown
Display Name: Unknown
Is SSD: false
VSAN UUID: ########-########-####-####-####-########226a
VSAN Disk Group UUID:
VSAN Disk Group Name:
Used by this host: false
In CMMDS: false
On-disk format version: -1
Deduplication: false
Compression: false
Checksum:
Checksum OK: false
Is Capacity Tier: false
Encryption Metadata Checksum OK: true
Encryption: false
DiskKeyLoaded: false
Is Mounted: false

   Creation Time: Unknown

Alternatively, use the following command to list the Disk mappings on the Host to identify the failed disk:

If the disk is listed with a UUID instead of an Disk identifier (e.g. naa.####), this indicates that vSAN has failed out the Disk, as shown in the example below:

[root@####:~] vdq -iH
Mappings:
DiskMapping[0]:
SSD: naa.58ce########fec5
MD: naa.58ce########a7f9
MD: naa.58ce#######bbd1
MD: naa.58ce#######02a5
MD: naa.58ce########9d69
MD: naa.58ce########aaf5
MD: naa.58ce########a7e5
MD: ########-########-####-####-####-########226a (This is a masked UUID)

3.) To identify the Display name (= Disk identifier) of the failed disk:

Remark: Will only work if the logs have not rolled over already

grep <UUID> /var/log/vmkernel.log

Example output:

YYYY-MM-DDTHH:MM:SS.ZZ cpu0:7053521)LSOM: LSOMLogDiskEvent:7509: Disk Event permanent error propagated for MD ########-########-####-####-####-########226a (naa.58ce######aad9:2)

4.) On the affected vSAN Host: Remove the Disk Group using the "No Data Migration" option:
Based on Example outputs above:

esxcli vsan storage remove -u ########-########-####-####-####-########226a

Syntax:

esxcli vsan storage remove -u <Disk-UUID> -m noAction

-u <Disk-UUID> Specified the UUID of the Disk Group

-m noAction      This is the CLI equivalent of "No Data Migration" option. It tells vSAN to delete the Disk group without evacuating the data to the other vSAN Hosts

Remarks:

It is critical that the correct UUID is removed. If the wrong UUID is removed this could cause data loss
While for any actions on Disks it is preferred to put a vSAN Host in Maintenance Mode first, the nature and/or the current status of the Disk failure, might result in the fact that attempting to put the Host into Maintenance prior removing the Disk Group will hang or error out
If the Disk removal task fails, reboot the vSAN Host and try again. Always attempt to place the vSAN Host into Maintenance Mode with "Ensure Accessibility" first.
If the ESXi host cannot enter Maintenance Mode normally, you must force it into Maintenance Mode using the "No Data Migration" option.

5.) Proceed with physically replacing the failed Disk

5.1) If the Host is not in Maintenance Mode: Put the Host into Maintenance Mode with "Ensure Accessibility"

5.2) Sut down the Host and physically replace the failed Disk(s)

5.3) Power the Host back on

6.) Recreate the Disk Group by using the vSAN Disk Management in vCenter

If assistance is required, open a Ticket with VMware by Broadcom Support.

Additional Information

If needed, retrieve Path/Slot information for the failed Disk to assist with HW identification:

Display Path Information:

Syntax:

esxcfg-mpath -bd <naa.disk identifier device>

Based on the example outlined in the Section "Resolution" of this KB Article:

esxcfg-mpath -bd naa.58ce########aad9
naa.58ce########d9 : VMware Serial Attached SCSI Disk (naa.58ce########aad9)
vmhba1:C0:T1:L0 LUN:0 state:active sas Adapter: 5005########8c11 Target: 5000########02af

The device is target #1 on vmhba1.

Display Slot/Physical Information:

Syntax:

esxcli storage core device physical get -d <naa identifier device>
esxcli storage core device raid list -d <naa identifier device>

Based on the example outlined in the Section "Resolution" of this KB Article:

esxcli storage core device physical get -d naa.58ce########aad9
Physical Location: enclosure 2, slot 5

esxcli storage core device raid list -d naa.58ce########aad9
Physical Location: enclosure 2, slot 5

Note: The above commands may not work with certain drivers as the vSAN disk serviceability plugin is not coded for all Drivers. Current supported drivers:

hpsa
nhpsa
iavmd
nvme_pcie
lsi_mr3
lsi_msgpt3
lsi_msgpt35
smartpqi

If the Driver is not listed (= supported), contact Hardware Vendor ( Hardware Vendor needs to work in their side to engage Broadcom Engineering for adding driver )

Following some examples indicating the Driver is not supported (= System cannot interact with the device to retrieve the required information or control the LED )

esxcli storage core device physical get -d naa.58ce########aad9
Unable to get location for device naa.99ac########ff6: No LSU plugin can manage this device.

esxcli storage core device raid list -d naa.58ce########aad9
Unable to get location for device naa.62ff########eei3: Can not manage device!