The Host memory status reports red alerts in the vSphere Client on HPE servers

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information when upgrading to ILO Firmware version to 2.30 on HPE ProLiant Gen10 and Gen10 Plus Servers which has multi LUN enabled where Mem_Stat_* sensors are reported on LUN 1. And eventually will see the Red alerts for these sensors in vSphere Client.

Symptoms:
On HPE ProLiant Gen10 and Gen10 Plus Servers running VMware ESXi 6.5.x, 6.7.x and 7.0.x with HPE Integrated Lights-Out 5 (iLO 5) Firmware Version 2.30 which has multi LUN enabled, you experience these symptoms:

The Memory Sensor Status reports Red alerts in the vSphere Web Client.
The IPMI SDR list dump on the ESXi host provides the following output to identify the issue.

For example:

$localcli hardware ipmi sdr list -p

40 0.2 Memory Device 2 Mem_Stat_C01S03 32.2 Configuration Error sensor-discrete 192 Memory 2020-09-21T00:46: 40 a1 00 51 02 2b 20 01 02 20 02 03 c2 0c 6f 93 00 00 00 d3 00 c0 00 00 00 00 00 00 00 00 00 00 cf 4d 65 6d 5f 53 74 61 74 5f 43 30 31 53 30 33 00
41 0.3 Memory Device 3 Mem_Stat_C01S04 32.3 Configuration Error sensor-discrete 192 Memory 2020-09-21T00:46: 40 a2 00 51 02 2b 20 01 03 20 03 03 c2 0c 6f 93 00 00 00 d3 00 c0 00 00 00 00 00 00 00 00 00 00 cf 4d 65 6d 5f 53 74 61 74 5f 43 30 31 53 30 34 00
42 0.4 Memory Device 4 Mem_Stat_C01S05 32.4 Configuration Error sensor-discrete 192 Memory 2020-09-21T00:46: 40 a3 00 51 02 2b 20 01 04 20 04 03 c2 0c 6f 93 00 00 00 d3 00 c0 00 00 00 00 00 00 00 00 00 00 cf 4d 65 6d 5f 53 74 61 74 5f 43 30 31 53 30 35 00
43 0.5 Memory Device 5 Mem_Stat_C01S06 32.5 Configuration Error sensor-discrete 192 Memory 2020-09-21T00:46: 40 a4 00 51 02 2b 20 01 05 20 05 03 c2 0c 6f 93 00 00 00 d3 00 c0 00 00 00 00 00 00 00 00 00 00 cf 4d 65 6d 5f 53 74 61 74 5f 43 30 31 53 30 36 00
44 0.10 Memory Device 10 Mem_Stat_C01S11 32.10 Configuration Error sensor-discrete 192 Memory 2020-09-21T00:46: 41 a9 00 51 02 2b 20 01 0a 20 0a 03 c2 0c 6f 93 00 00 00 d3 00 c0 00 00 00 00 00 00 00 00 00 00 cf 4d 65 6d 5f 53 74 61 74 5f 43 30 31 53 31 31 00
The IPMI sdr list for the Mem_Stat_* sensors above indicate that these sensors are reported on LUN 1 (For example: 7th byte - 01) in the readings "a1 00 51 02 2b 20 01….” along with the “Configuration Error” reported under “Computed Reading”. In this case the Web Client reports these sensors as Red alerts.

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

This issue occurs after upgrading to ILO firmware version to 2.30 on HPE ProLiant Gen10 and Gen10 Plus Servers where LUN 1 was enabled and Mem_Stat_* sensors were reported on LUN 1. The sensors reported on LUN 1 were not decoded appropriately by the Hardware health monitoring system.

Resolution

The fix is in iLO firmware 2.31 version as per HPE advisory.

VMware has resolved the issue of LUN address reading in:

VMware ESXi 6.7 Patch release ESXi670-202011002
VMware ESXi 7.0 Update 1c
VMware ESXi 6.5 Patch release ESXi650-202102001

Workaround:
To work around this issue, Hewlett Packard Enterprise has released a Customer Advisory. For more information, see:

Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third- party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

Alternatively, disable the cimsvc plug-in.

Note: This also disables Hardware Health monitoring.

To disable the cimsvc plug-in:

Log in as root through SSH.
Run this command:

/etc/init.d/hostd stop
Edit the /etc/vmware/hostd/config.xml file and modify line <enabled>true</enabled> to <enabled>false</enabled>.

For example:

<cimsvc>
<path>libcimsvc.so</path>
<enabled>true</enabled>
</cimsvc>
Run this command:

/etc/init.d/hostd start

Additional Information

Impact/Risks:
Disables Hardware Health monitoring.