System Disk Usage Percentage Increasing (system.disk.persistent.percent) - OMS Agent

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

Comparing an identical but unaffected foundation to the foundation with high root partition disk usage, we discovered that the Operations Management Suite (OMS) Agent is not cleaning up old zip files for a total of 854 MB. Subtracting 854 MB from the disk space used on the affected foundation, which was 2300 MB, we get 1446 MB, which is the space used on the identical but unaffected foundation that does not use the Operations Management Suite (OMS) Agent.

Environment

Cause

The Microsoft monitoring tool, Operations Management Suite (OMS) Agent, is filling up the root partition by not purging zip files after each update. This is a Microsoft bug.

Resolution

Step 1

Check to see if left over zip files exist. Running the following command on an affected VM will reveal the zip files and confirm if OMS Agent is filling up the root partition.
Option -x, --one-file-system - skip directories on different file systems. We want to interrogate the root.

$ du -axh / | sort -rh | head -20

In the screenshots below, the comparison described above is used to show the difference between an affected and unaffected VM. Notice the zips which exist on the left side but not on the right side. The sum of the zips is the exact difference in disk usage between the two.

Step 2

• Option 1. Vertically scale the VMs to increase disk size
• Option 2. Purge the old zip files (verify with the Azure support team if this is safe)
• Option 3. Report the issue with Microsoft Azure Support.