vIDM 3.3.6 | Root partition fills up rapidly on one of the three nodes

Products

• VMware Aria Suite

Issue/Introduction

Impacted Versions:

vIDM versions 3.3.6

Symptoms:

Issue Description:
The root partition fills up rapidly on vIDM 3.3.6. The main reason behind this is:

1. Enormous /var/log/messages files.
2. Enormous /var/log/journal files.

Symptoms:

• Failures during the startup of services such as Horizon-Workspace, Elasticsearch, and RabbitMQ.
• Empty config-state files on VIDM nodes.

Environment

VMware Identity Manager 3.3.x

Cause

This is a known issue in VIDM v3.3.6 

Resolution

This issue has been fixed in VIDM 3.3.7 

Workaround:

Manual Remediation Steps:

To manually address this issue, follow these steps:

1. SSH into the appliance as the root user.

2. Remove all the journal logs (/var/log/journal/* except the symlink)
   a. remove the /var/log/messages
   b. run logrotate /etc/logrotate.conf 
   c. Modifiy the /etc/logrotate.conf from weekly rotation to daily rotation
   d. Add this to the /etc/cron.d/hzniptables file
   */1 * * * * cat /dev/null >/var/log/messages

   If such disk space is encountered again:

   Do
   a. logrotate /etc/logrotate.conf
   b. Add the above entry to the hzniptables
   c. modify the /etc/logrotate.conf entry of "weekly" to "daily"

Steps on how to clear issue related to syslog 

1. First clear any pending open files running 
   lsof -n | grep delete
2. Then restart rsyslog
   systemctl restart rsyslog

 

 

 

 

Additional Information

Note:

• In a cluster setup, execute these steps on all nodes to ensure complete resolution.
• We also might need to reboot the appliance once we have the workaround in place