VMware Identity Manager 3.3.x stuck in boot loop due to 100% full root partition

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

A VMware Identity Manager appliance may enter a continuous boot loop after attempts to modify disk partitions or when the root partition reaches 100% capacity. This is often observed when performing maintenance while the appliance has high numbers of unassigned shards or existing OpenSearch health issues. You will be unable to access the UI or SSH into the management interface.

Environment

• Product: VMware Identity Manager
• Version: 3.3.x
• OS: Photon OS

Cause

The root partition has reached 100% utilization. This prevents the operating system from creating necessary temporary files during the boot sequence, leading to a kernel or service failure that triggers a restart.

Resolution

Note: Before proceeding, ensure you have a valid snapshot or backup of the node.

1. Access Single-User Mode:
   • Restart the appliance from the vCenter console.
   • At the Photon OS splash screen, you press e to enter the GNU GRUB edit menu.
   • Locate the line beginning with linux and you add rw init=/bin/bash to the end of the line.
   • Press F10 to boot into the bash shell.
2. Remount File System:
   • Run the following command to make the root partition writable:

     mount -o rw,remount /

3. Identify and Clear Space:
   • Check disk usage: df -h.
   • Navigate to log directories (e.g., /var/log) and identify large or unnecessary log files.
   • Delete files to free up space (e.g., rm <filename>).
4. Finalize and Restart:
   • Once space is cleared, restart the appliance:

     reboot -f

Additional Information

If the root account is also locked due to disk space issues, you can reset it while in single-user mode using the passwd command or unlock it using pam_tally2 -u root --reset.