Excessive Hardware health alarms being triggered for "Sensor -1 type" on ESXi hosts running vSphere 6.7/6.5

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides information and workaround for the hardware health alarms getting triggered (for sensor -1 type) on ESXi hosts

Symptoms:
After upgrading the ESXi host(s) to 6.7/6.5 Update 3, you see these symptoms:

Logging in to the vCenter Server appliance (VCSA) using the vSphere Web Client fails.
You see the error similar to:

503 Service Unavailable
If you are able to log in to VCSA using the vSphere Web Client, in the vCenter Events window, you see entries similar to:

Sensor -1 type , Description Battery 7.1 state assert for . Part Name/Number N/A N/A Manufacturer N/A
Sensor -1 type , Description CPU1 Level-1 Cache state assert for . Part Name/Number N/A N/A Manufacturer N/A
Sensor -1 type , Description CPU1 Level-2 Cache state assert for . Part Name/Number N/A N/A Manufacturer N/A
Sensor -1 type , Description CPU1 Level-3 Cache state assert for . Part Name/Number N/A N/A Manufacturer N/A
In the vSphere Web Client > vCenter Server > Cluster > Host > Monitor > Events, you see entries similar to:

Alarm 'Host hardware sensor state' on esx.abcd.com. triggered by event 14306564 'Sensor -1 type , Description Disk 7 on HPSA1 : Port 2I Box 3 Bay 7 : 3576GB : Unconfigured Disk : OK state deassert for . Part Name/Number N/A N/A Manufacturer N/A'

Sensor -1 type , Description Battery 7.1 state assert for . Part Name/Number N/A N/A Manufacturer N/A

Note: The log excerpts above are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.7.x

Resolution

This issue is resolved :
ESXi 6.7 Patch Release ESXi670-201911001.
ESXi 6.5 Patch Release ESXi650-202110001.

Workaround:
To work around this issue if you do not want to patch the ESXi host:

You can avoid this issue by disabling the WBEM services. To disable the WBEM services on the ESXi host, run this command:

esxcli system wbem set --enable false

To re-enable the CIM agent on the ESXi host, run this command:

esxcli system wbem set --enable true

Note: When an ESXi host is rebooted and if a 3rd party providers are present, the WBEM service will become enabled on reboot. In this instance, the step of disabling WBEM services will have to be repeated. However, this can be avoided by turning off the sfcbd-watchdog in chkconfig.

To turn off sfcbd-watchdog from chkconfig, run this command:

chkconfig sfcbd-watchdog off

Note: For more information, see enabling or disabling WBEM services.

If you do not want to disable the WBEM services, you may continue to receive email alerts and events spew. To workaround this issue, reconfigure the hardware sensor alarm to restrict the emails for these specific events:

Log in to the vCenter Server using the vSphere Web Client.
Select the vCenter Server and click Configure.
Click Alarm Definitions.
Select the Alarm Name Host hardware sensor state.
Click Edit > Next.
In the Alarm Rule section, click Add Argument.
Select argument as sensorNumber and select operator as does not start with and provide sensor number as -1 which is causing alarm to trigger the alert.
Click Next and Save.

Notes:

These steps should substantially limit the excessive events spew for alarms and hence slowdown the database growth in the environment. However, the events for hardware sensor will continue to trigger.
If you do not want to disable WBEM services, you may still see the VCSA SEAT partition growing (resulting in the vCenter Server database size growth) due to these events. For more information, see Determining where growth is occurring in the vCenter Server database (1028356).
For the purpose of identifying where growth is and procedure for truncating tables occurring in MSQL and Oracle databases, refer to your vendor documentation.
For more information on increasing disk space on the vCenter Server appliance, see Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5 and 6.7 (2145603).

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
For additional information, see Monitor Database Use.

Impact/Risks:

The functional impact should be minimal as the alarms getting triggered (for Sensor -1 type) are not signifying any issue with the hardware. But these periodic alarms can result in excessive emails alerts if the user subscribed to email notification for Host hardware sensor state alarm in vCenter.
Disabling the wbem service will not produce any negative impact, but the hardware health status will not be available.
Additionally, these events can result in the VCSA database increasing in size leading to issufficient disk space issues as well as the SEAT partition size going above the 95% threshold which causes the VCSA to become inaccessible through the vSphere Web Client displaying the error: 503 Service Unavailable.