Identity Manager root partition is full on one or more nodes in the cluster

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The root partition (/dev/sda4) is completely full (100%) on one of the VIDM nodes in the cluster

root@<VIDM-FQDN> [ ~ ]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       7.9G     0  7.9G   0% /dev
tmpfs                          7.9G   12K  7.9G   1% /dev/shm
tmpfs                          7.9G  808K  7.9G   1% /run
tmpfs                          7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda4                       17G  16G    0   100% /
tmpfs                          7.9G  184K  7.9G   1% /tmp
/dev/sda2                      119M   26M   87M  23% /boot
/dev/mapper/db_vg-db            20G  502M   19G   3% /db
/dev/mapper/tomcat_vg-horizon   20G  2.8G   16G  15% /opt/vmware/horizon
tmpfs                          1.6G     0  1.6G   0% /run/user/1001
tmpfs                          1.6G     0  1.6G   0% /run/user/0

The directory sync operation (Navigate to Identity & Access Management > Directories > Sync now) is failing with the following error.

Failed to save config to disk

Login attempts to Aria Operations using Single Sign-On (SSO) fail with the error "Username or password is incorrect".

Environment

VMware Identity Manager 3.3.x
VMware Aria Operations 8.18.x

Cause

The root partition (/dev/sda4) on one or more Identity Manager nodes is full. This is often caused by unlinked log files that remain in use by rsyslog or excessive accumulation of journal, message and audit files. When the partition is full, the system cannot update critical files such as config-state.json.

Resolution

IMPORTANT:

First take a snapshot of all appliances in the vIDM cluster (non-memory, quiesced) before proceeding any further as per KB How to take a Snapshot of VMware Identity Manager.
If extending the disk space per option 4, delete any existing snapshots for the vIDM appliances, and create clones of the appliances instead for any rollback purposes.

Option 1: Rotated log files still opened by rsyslogd

Rotated log files that are renamed, while unlinked from the filesystem, are sometimes still in use by rsyslog.

Log in to the affected Identity Manager node as root.

Check for deleted log files still held open by the system:

lsof +L1 | grep delete | grep "rsyslogd"

Example of output of above command is:

rsyslogd   512314       root   15w   REG    8,4 97697061     0 196612 /var/log/messages-1770865801 (deleted)
rsyslogd   512314       root   97w   REG    8,4 97697061     0 196612 /var/log/messages-1770865801 (deleted)

Restart the rsyslog service to release these file descriptors and reclaim space:
```
systemctl restart rsyslog
```
Monitor the usage of the root partition with:
```
df -B M | grep -iE "filesystem|sda4"
```
If disk space is still low, choose another option below.

Option 2: Clearing journal files

Remove the journal files with rm -rf /var/log/journal/*
Remove file messages file with rm -rf /var/log/messages
Check if audit log is consuming space.
1. Change Directory: cd /var/log/audit
2. Check the size using: do ls -lh
3. If audit log file is being too large then truncate (clean the content without deleting the file) the audit.log file with this command:
```
truncate -s 0 audit.log
```
Modify the /etc/logrotate.conf from weekly rotation to daily rotation and then run command logrotate /etc/logrotate.conf
Add this to the /etc/cron.d/hzniptables file. The hzniptables file is present under /etc/cron.d
```
*/1 * * * * cat /dev/null >/var/log/messages
```
Add/Modify the following lines to /usr/local/horizon/conf/runtime-config.properties on each node:
```
analytics.deleteOldData=true
analytics.maxQueryDays=90
```
Edit vi /etc/rsyslog.conf(and remove all the input methods) after confirming that /usr/local/horizon/scripts/enableRSyslog.hzn status shows no syslog present. (Copy backup before edit)
Restart the service on the nodes, as after updating the rsyslog.conf restart is required for changes to take affect.
```
ps aux | grep rsyslog
systemctl status rsyslog
systemctl restart rsyslog
```

Option 3: Clean up pgService backup files

If numerous *.backup files exist for /var/log/pgService/, remove them with the following command per KB 372820:

rm -f /var/log/pgService/*.backup

Option 4: Resize / partition of the appliances

Resize the /dev/sda4 disk to 20G on all appliances of the cluster by following the steps detailed in this KB: How to Increase vIDM appliance disk space (broadcom.com)

Option 5: Investigate disk usage

Check the output of below config file to look for the configuration:
```
cat /etc/logrotate.conf
cat /etc/cron.d/hzniptables
```
Note: Check the result of the both the commands again, so that we could compare and check which directory has consumed more space.
Exclude the /opt/vmware and /db directories so that we just focus on root directory:
1. Run the below command to check disk usage:
```
du -ah / 2>/dev/null --exclude=/opt/vmware/horizon --exclude=/db | sort -rh | head -n 20
```
2. Check the disk usage of /var folder:
```
du -ah /var 2>/dev/null | sort -rh | head -n 20
```
  Note: Observe any dip in the Partition Utilization metric or is it still incrementing? You can check the output of above commands for next couple of days more.
  You also can reboot the cluster and compare before and after reboot output.
Check all config-state.json file over all 3 nodes and check /etc/systemd/journald.conf for SystemMaxUse property (by default 100M).
1. Check the cache usage with this command: du -sh /var/cache/ if normal in MB size
2. Review the list of opened files which are deleted but are hung due to some issue: lsof | grep '(deleted)'
3. Run this command view /etc/rsyslog.conf where the input methods remove instead of cache. Then change it on a single node to monitor if this resolves the issue, Apply the same changes to the other nodes after few days.
4. Check the frequency of the Directory Sync,
  Note: If set to hourly then check users are getting synced every 1hr. Also it is recommend to change the sync connector for one of the directories having max Users and see if the other node is also having similar impact on disk space as on another node.
Check if the hardware requirement matches the guidelines: VIDM Guideline
Run the below query and check the csv file result before updating the directory sync frequency:
1. Connect to the database:
```
cat /usr/local/horizon/conf/db.pwd
psql -U postgres saas
```
2. Run this statement to export:
```
copy (select * from "CacheEntry") to '/tmp/CacheEntry.csv’ with csv;
```
  Note: Can run the same above command again after 2-3 days and compare the result.
Compare below command output for all 3 nodes:

Before setting the sync frequency to manual and after setting the sync frequency to manual
You may try after a few days / 2-3 days of soak time require to have the significant difference in the disk size.

Commands to investigate disk usage:
```
du -ah / 2>/dev/null | sort -rh | head -n 100
find ./ -type f -size +100M | less
find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 >  100' | sort -k2,2nr |less
```

Additional Information

If the required space is still not getting reclaimed, follow the KB below to check the gc.logs: Uncompressed gc.logs are causing the /root partition to run out of space in VMware Identity Manager.

Note 1: The nodes can experience higher load due to misconfiguration of the connectors. You can distribute the load across all nodes and check for improvement in disk utilization.
Note 2: Reboot the nodes so that jvm and other CPU processes release temporary files.
Note 3: Validate the KB (HW-134096 - VMware Identity Manager Connector may fail to communicate due to config-state.json corruption (broadcom.com)) to restore the configuration files if they are corrupt or empty.