The root '/' partition continually fills up with 'ghost; data in vIDM.

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

In a vIDM environment, the root '/' partition continually fills up over a period of time.
Increasing the root partition size only delays the issue.
Performing a cluster reboot resolves the storage utilisation.
Running the command below, shows the 'rsyslog' has large, deleted files locked.
- ```
lsof +L1 | grep '(deleted)'
```

Environment

VMware Identity Manager 3.3.7

Cause

The rsyslog service retains file handles for very large log files that have already been deleted. Because these files are still held open by the active 'rsyslog' process, the operating system cannot free the associated disk space on the root partition.

Resolution

To immediately removed the lock, restart the 'rsyslog' service;

```
systemctl restart rsyslog
```

To prevent the issue from reoccurring, follow the steps below;

Verify that no active syslogs are currently configured by checking the status:
```
/usr/local/horizon/scripts/enableRSyslog.hzn status
```
Once you confirm no syslogs are present, create a backup of the current 'rsyslog' configuration:
```
cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
```
Open the configuration file in a text editor:
```
vi /etc/rsyslog.conf
```
Comment out all input methods from the file, then save and exit.
Restart the rsyslog service to apply the configuration changes:
```
systemctl restart rsyslog
```
Verify the service is running correctly:
```
systemctl status rsyslog
```

Additional Information

1. The steps listed above have were found in a larger KB related to mitigating issues with storage on the root partition;

Identity Manager root partition is full on one or more nides in the cluster.

2. After edit rsyslog.conf, If rsyslog status systemctl status rsyslog showing error during parsing file /etc/rsyslog.conf,
you can validate the error with command rsyslogd -N1.

: error during parsing file /etc/rsyslog.conf, on or before line 189: warnings occurred in file '/etc/rsyslog.conf' arou
nd line 189 ........
: invalid character in selector line - ';template' expected ..........
:error during parsing file /etc/rsyslog.conf, on or before line 189: errors occurred in file '/etc/rsyslog.conf' around
line 189 ........

Verify 189 line with sed -n '185,195p' /etc/rsyslog.conf

local0,local1.* /var/log/localmessageslocal2,local3.* /var/log/localmessageslocal4,local5.* /var/log/localmessageslocal6,local7.* /var/log/localmessageslocal8.* /var/log/boot.log#### Pgpool Configuration startlocal1.* /var/log/pgService/pgService.log#### Pgpool Configuration end

* Note: There is no such thing as local8 in the syslog standard. Because rsyslog doesn't recognize local8 as a valid keyword, it completely fails to parse the line, assumes the configuration is corrupt, and throws that invalid character in selector line. The system only supports eight of them, numbered from local0 up to local7.

To fix the error Add a # at the very beginning of the local8 line. and save and restart the rsyslog file.

systemctl status rsyslog

Verify the output only shows those two harmless warning but no errors, and syntax is finally clean!