VMware Identity Manager node fails to boot resulting in "no healthy upstream" error during VMware Aria Automation authentication

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware vRealize Automation 8.x

Issue/Introduction

When attempting to authenticate to VMware Aria Automation, users encounter a "no healthy upstream" error message.

Upon investigation, one or more nodes in the VMware Identity Manager cluster may be unresponsive and fail to boot. Observing the node via the virtual machine console reveals that the appliance hangs during the startup sequence. Once the operating system is forced to boot, checking the disk space indicates that the /db partition is at 100% utilization.

Environment

• VMware Identity Manager 3.3.7

• VMware Aria Automation

• VMware Aria Suite Lifecycle (LCM)

Cause

The immediate boot failure is caused by filesystem inconsistencies that prevent the operating system from starting.

The underlying cluster and authentication failure is caused by the /db partition reaching 100% capacity. When this partition is completely full, the underlying PostgreSQL database cannot execute necessary read and write transactions. This breaks the database cluster synchronization across nodes, causing authentication requests from upstream applications (such as VMware Aria Automation) to fail.

Resolution

• Access the unresponsive VMware Identity Manager node via the virtual machine console.

• Restart the appliance and interrupt the boot process to access the GRUB menu.

• Append the following parameter to the boot sequence to automatically repair the filesystem inconsistencies: fsck.repair=yes per Knowledge Base Article 326323

• Allow the operating system to successfully boot and come online.

• Increase the size of the /db partition from the default 20GB to 40GB to accommodate database operations. Follow the exact steps outlined in Knowledge Base Article 319356.

• Once the partition resize is complete, log into VMware Aria Suite Lifecycle (LCM).

• Execute a Remediate operation on the VMware Identity Manager cluster. This will repair and synchronize the PostgreSQL database cluster, returning it to a healthy state.