vIDM 3.3.6 or lower version | Root partition fills up rapidly on one of the three nodes
search cancel

vIDM 3.3.6 or lower version | Root partition fills up rapidly on one of the three nodes

book

Article ID: 345391

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Impacted Versions:

vIDM versions 3.3.6 or earlier.


Symptoms:

Issue Description:
The root partition fills up rapidly for 3.3.6 or lower vIDM version. The main reason behind this is:

  1. Enormous /var/log/messages files.
  2. Enormous /var/log/journal files.

Symptoms:

  • Failures during the startup of services such as Horizon-Workspace, Elasticsearch, and RabbitMQ.
  • Empty config-state files on VIDM nodes.



Environment

VMware Identity Manager 3.3.x

Cause

This is a known issue in VIDM v3.3.6 and below.

Resolution

This issue has been fixed in VIDM 3.3.7 and later.


Workaround:

Manual Remediation Steps:

To manually address this issue, follow these steps:

  1. SSH into the appliance as the root user.
  2. Remove all the journal logs (/var/log/journal/* except the symlink)
    b. remove the /var/log/messages
    c. run logrotate /etc/logrotate.conf 
    d. Modifiy the /etc/logrotate.conf from weekly rotation to daily rotation
    e. Add this to the /etc/cron.d/hznIptables file
    */1 * * * * cat /dev/null >/var/log/messages

    If such disk space is encountered again:

    Do
    a. logrotate /etc/logrotate.conf
    b. Add the above entry to the hznIptables
    c. modify the /etc/logrotate.conf entry of "weekly" to "daily"

 

 

 

 

Additional Information

Note:

  • In a cluster setup, execute these steps on all nodes to ensure complete resolution.
  • We also might need to reboot the appliance once we have the workaround in place