Automation deployments fail in Aria Automation with error: "Unable to process request. Status code: 403" due to RaaS service failure

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Automation deployments in Aria Automation fail due to SaltStack-related issues, with deployment requests returning a 403 Forbidden error and the RaaS service unable to restart, thereby impacting automation workflows and minion communication.
• Day 1 and Day 2 actions in Aria Automation fail with the error: Unable to process request. Status code: 403, Response 403: Forbidden (Dynamic Script Module name: Saltservice#27).
• Attempts to restart the RaaS service using "systemctl restart raas" fail with the error: [Errno 28] No space left on device: '/var/lib/raas/run/metrics/histogram_###.db'.

Environment

VMware Aria automation 8.x

Aria Automation Config 8.x

Cause

The root filesystem on the RaaS instance reached 100% utilization, which prevents the RaaS service from writing metric database files under /var/lib/raas/run/metrics/. As a result, the RaaS service fails to restart and automation deployments are unable to proceed.

Resolution

1.  Verify the disk usage using df -h and confirm the root partition is over utilized.
2. Identify the large log directories by running du -sh * under /var/log and review the directories that are consuming significant disk space. 
3. Clear logs from the identified logs directories to free up disk space. For example, if the identified directories are /var/log/messages & /var/log/raas, run below echo command to empty the log file.

   echo > /var/log/messages
   echo > /var/log/raas

4. Stop and restart the raas service with following commands:

   systemctl stop raas
   systemctl start raas

5. Retry the day1/day2 task from the Aria automation to confirm the issue is resolved.