"Disk Exhaustion" alarm repeatedly triggers every 2 minutes on VCSA 7.0/7.0 U1/7.0 U2

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• You see "Disk Exhaustion" alarms being repeatedly triggered, along with flapping status change from 'Green' to 'Yellow' and 'Yellow' to 'Green' at the same time.
• In /var/log/vmware/applmgmt/StatsMonitor.log , you may see the below entries:

YYYY-MM-DD info StatsMonitor[01656] [Originator@6876 sub=StatsMonitor] eventMessage Size: 2090 Batch Id: 3
YYYY-MM-DD info StatsMonitor[01656] [Originator@6876 sub=StatsMonitor] eventMessage: [{"event_id": "vim.event.ResourceExhaustionStatusChangedEvent","args":{"resourceName":"storage_util_filesystem_seat","oldStatus":"yellow","newStatus":"green","reason":" "}},{"event_id": "vim.event.ResourceExhaustionStatusChangedEvent","args":{"resourceName":"storage_util_filesystem_db","oldStatus":"yellow","newStatus":"green","reason":" "}},{"event_id": "vim.event.ResourceExhaustionStatusChangedEvent","args":{"resourceName":"storage_util_filesystem_vtsdb","oldStatus":"yellow","newStatus":"green","reason":" "}}]
YYYY-MM-DD info StatsMonitor[01656] [Originator@6876 sub=StatsMonitor] Post event command and args are ready, invoking ...
YYYY-MM-DD info StatsMonitor[01656] [Originator@6876 sub=StatsMonitor] Post alarm failed: Event vim.event.ResourceExhaustionStatusChangedEvent successfully posted to http
://localhost:8085/sdk
--> Event vim.event.ResourceExhaustionStatusChangedEvent successfully posted to http://localhost:8085/sdk
--> Event vim.event.ResourceExhaustionStatusChangedEvent successfully posted to http://localhost:8085/sdk
--> Event vim.event.ResourceExhaustionStatusChangedEvent successfully posted to http://localhost:8085/sdk

--> Failed to post event vim.event.ResourceExhaustionStatusChangedEvent to http://localhost:8085/sdk. Err: unrecognized vSphere resource name "storage_util_filesystem_lifecycle".
-->
--> Event vim.event.ResourceExhaustionStatusChangedEvent successfully posted to http://localhost:8085/sdk
--> Failed to post event vim.event.ResourceExhaustionStatusChangedEvent to http://localhost:8085/sdk. Err: unrecognized vSphere resource name "storage_util_filesystem_vtsdb".
--> /bin/statsmonitor_alarm_dispatcher.py:34: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
--> import imp
--> Warn: Events were sent to only 0 vpxd conns.
-->
YYYY-MM-DD info StatsMonitor[01656] [Originator@6876 sub=StatsMonitor] Exit post event ...

 

Environment

VMware vCenter Server 7.x

Cause

Some alarm definitions are missing from /bin/statsmonitor_alarm_dispatcher.py , which prevents StatsMonitor from posting vim.event.ResourceExhaustionStatusChangedEvent successfully.

Resolution

This issue is resolved in vCenter Server 7.0 U3. You can download from Customer Connect Downloads page.

Workaround:
To workaround this issue, please replace the statsmonitor_alarm_dispatcher.py on the VCSA with the attached file on this article:

1. Download the attached statsmonitor_alarm_dispatcher.py
2. Backup/rename the current statsmonitor_alarm_dispatcher.py file through the below steps:
   1. Connect to the VCSA through SSH
   2. Run the below command to navigate to /bin

# cd /bin

3. Run the below command to backup and rename the current  statsmonitor_alarm_dispatcher.py file.

# mv statsmonitor_alarm_dispatcher.py statsmonitor_alarm_dispatcher.py.bak

3. Using WINSCP, copy the statsmonitor_alarm_dispatcher.py to the VCSA virtual machine in the /bin directory.

Note: If you got the below error while connecting to the VCSA, see Error when uploading files to vCenter Server Appliance using WinSCP (2107727)

Host is not communicating for more than 15 seconds. If the problem repeats, try turning off 'Optimize connection buffer size'