Directory Services synchronisation failing due to high root partition utilisation in vIDM

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VMware Identity Manager (vIDM) experiences Directory Services synchronization failures on a specific domain.
• Testing the bind connection results in a continuous spinning wheel.
• The vIDM UI Health diagnostic tab indicates that cluster nodes are not reporting health status.
• Telnet connections to the lDC FQDN on port 636 are successful.
• 'df -h' shows high utilisation (80-100%) of the root partition.
• 'lsof +L1 | grep '(deleted)'' shows that 'rsyslog' had locked large deleted files.

Environment

VMware Identity Manager (vIDM) v3.3.7

Cause

The 'rsyslog' service locks deleted files, creating "ghost" files that completely exhaust the available space on the root (/) filesystem, leading to cluster instability and directory sync failures.

Resolution

If the output of the 'lsof +L1 | grep '(deleted)'' shows 'rsyslog' has locked files, the following steps can be taken to immediately release the locked files.

1. Open an SSH session to the affected vIDM node(s).

2. Restart the rsyslog service to release the locked deleted files by running the following command: systemctl restart rsyslog

3. Restart the horizon-workspace service to restore application health: service horizon-workspace restart

4. Verify that the root partition utilization has dropped using df -h and that the nodes are reporting healthy in the UI.

Additional Information

To prevent this issue in the future, the steps in the resolution of the KB below can be followed:

• The root '/' partition continually fills up with 'ghost' data in vIDM