Aria Automation node marked unhealthy due to disk usage health check failure and Load Balancer stops routing traffic to the affected nodes.

Products

• VCF Automation

Issue/Introduction

• Aria Automation nodes are being marked as unhealthy by the Load Balancer due to the /opt/health/run.sh script returning an HTTP 500 status. This leads to service disruption as the Load Balancer stops routing traffic to the affected nodes and status of Aria Automation Virtual servers will show as "Degraded" 

• The health check script /opt/health/run.sh includes a disk-usage check. When disk space is less than 20% available ,  this check fails with the following error:

  Running check disk-usage  
make: *** [/opt/health/Makefile:67: disk-usage] Error 1

Environment

Aria Automation 8.x

Cause

The health check script /opt/health/run.sh includes a disk-usage check. When disk space is less than 20% available ,  this check fails with the following error:

Running check disk-usage  
make: *** [/opt/health/Makefile:67: disk-usage] Error 1

This issue occurs when at least one system health check returns an error.

As a result, the script returns a non-200 status code (HTTP 500), causing the Load Balancer to treat the node as unhealthy and stop routing traffic to it.

Resolution

Determine which storage partition is almost full:

1. Check free disk space on the affected Aria Automation node:
   • vracli disk-mgr
   • Or across all nodes at once:
     • vracli cluster exec -- bash -c 'hostname; vracli disk-mgr'
   • Any disk with Available: less than 20% is tripping the alarm
2. If any change is intended, please take snapshots of all Automation nodes as a precaution
3. Where the /home partition has filled up, log bundles can often be deleted from /home/root
4. For help with this or other partitions, please contact Broadcom Support and note this Article ID (403457) in the problem description.
   • For more information, see Creating and managing Broadcom support cases.

If it is not possible to clear unnecessary files, please see the section below to expand the Aria Automation disk.

 

If space cannot be cleared:

1. Power Off the Aria Automation Nodes through Aria Suite Lifecycle. 
2. Increase the disk space of Aria Automation Node on vCenter  -Increase VMware Aria Automation appliance disk space
3. Power On the Aria Automation Nodes through Aria Suite Lifecycle.

Additional Information

Validate the Disk Space with Validating VMware Aria Automation and Automation Orchestrator 8.x health