Certain HPE NVMe Solid State Drive models fails after 4700 hours of operation

Products

• VMware vSAN

Issue/Introduction

Symptoms:

HPE NVMe SSDs experience unexpected failures after 4700 hours of operation (i.e, 6 to 7 months).

vSAN Disk groups goes offline.

Impacted HPE NVMe SSD models:-

HPE Model Number	HPE SKU	HPE SKU DESCRIPTION
MO001600KWVUU	P13699-B21,   P13699-H21,   P13699-K21	HPE 1.6TB NVMe x4 Lanes Mixed Use SSD
MO001600KWVUU	P18140-B21	HPE DX 1.6TB NVMe x4 Lanes Mixed Use SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD
VO002000KWVVC	P13695-B21,   P13695-H21,   P13695-K21	HPE 2TB NVMe x4 Lanes Read Intensive SSD
VO002000KWVVC	P18145-B21	HPE DX 2TB NVMe x4 Lanes Read Intensive SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD
MO003200KWVUV	P13701-B21,   P13701-H21,   P13701-K21	HPE 3.2TB NVMe x4 Lanes Mixed Use SSD
MO003200KWVUV	P18142-B21	HPE DX 3.2TB NVMe x4 Lanes Mixed Use SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD
VO004000KWVUR	P13697-B21,   P13697-H21,   P13697-K21	HPE 4TB NVMe x4 Lanes Read Intensive SSD
VO004000KWVUR	P18149-B21	HPE DX 4TB NVMe x4 Lanes Read Intensive SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD
MO006400KWVVA	P13703-B21,   P13703-H21,   P13703-K21	HPE 6.4TB NVMe x4 Lanes Mixed Use SSD

 

Environment

VMware vSAN 6.x
VMware vSAN 7.x

Resolution

Upgrade the drive firmware to version 4ICRHPK3, which HPE has released to prevent this issue.

If the failure occurs, the SSD might not recover even after the firmware upgrade.

For more information and affected disks/server models see the below advisories:
Bulletin: (Revision) HPE NVMe Solid State Drives - Critical Firmware Upgrade Required for Certain HPE NVMe Solid State Drive Models to Prevent Drive Failure After 4700 Hours of Operation

Please engage VMware and HPE support if additional assistance is required to get the environment back into a healthy state.