Error: 404 Not Found and service failures on vIDM cluster after storage outage

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

A three-node VMware Identity Manager (vIDM) cluster experienced an unclean power-off due to a storage failure ( or SAN outage). After restoration, the following symptoms occur:

The vIDM UI is inaccessible, returning a 404 Not Found error at "https://<vIDM-FQDN>/SAAS/apps/".
The opensearch service fails to start on all nodes. (Power on nodes following steps in section vSphere Method Graceful Shutdown and Power On of a VMware Identity Manager PostgreSQL cluster)
There is a role mismatch where the pgpool Master and postgres Primary are on different nodes.(This can be checked following KB Troubleshooting VMware Identity Manager postgres cluster deployed through Aria Suite Lifecycle)
Status "quarantine" is observed for postgres database members in "show pool_nodes" command.

System time is out of sync across the cluster.
Aria Suite Lifecycle reports CRITICAL or WARNING health statuses for vIDM, and Auto Recovery is enabled for the vIDM component.

Environment

VMware Aria vIDM 3.3.7

VMware Aria Lifecycle 8.18.x

Cause

An unclean shutdown caused by storage failure led to database role inconsistency. Additionally, NTP synchronization failures may occur due to unstable network connectivity, preventing the cluster from maintaining a consistent system clock across all nodes.

Resolution

The following steps prioritize vIDM health restoration before re-engaging management via Aria Suite Lifecycle.

Important: If Auto Recovery is enabled in Aria Suite Lifecycle, disable it before proceeding. Re-enable it only after the cluster is healthy and an Inventory Sync is completed.

1. Sequence the Cluster Power-On
To resolve the 404 error and service hang:
1.1. Take a snapshot (without memory) of all three vIDM nodes.
1.2. Power off all three nodes.
1.3. Power them on in the following order: Primary Node first, followed by Secondary Nodes.

2. Verify Node Configuration
Ensure consistency across all nodes by verifying these files:
cat /etc/hosts
cat /usr/local/etc/pgpool.conf | grep backend_hostname

3. Synchronize System Time
Correct any time drift using the Horizon NTP script:
# Get current NTP status
/usr/local/horizon/scripts/ntpServer.hzn --status
# Set a specific NTP server if drift is detected
/usr/local/horizon/scripts/ntpServer.hzn --set "<ntp_server_ip>"
# Verify synchronization
timedatectl timesync-status

4. Recover Quarantined Nodes (Details in One or more nodes of VMware identity manager is in Quarantine state, alternatively, use the procedures in the KB 437705 under Additional Information.)
If a node shows a "quarantine" status in "show pool_nodes":
4.1. On the quarantined node: Stop the database service.
/etc/init.d/vpostgres stop
4.2. On the primary node: Force recovery of the affected node ID.
/usr/local/bin/pcp_recovery_node -h localhost -p 9898 -U pgpool -n <node_id> -W
4.3. On the quarantined node: Restart the service.
/etc/init.d/vpostgres start

5. Align pgpool mater and postgres primary role
To ensure the pgpool Master and postgres Primary reside on the same node:
5.1. Stop the pgpool service on all nodes:
/etc/init.d/pgService stop
5.2. Start the service on the postgres Primary first, then the Standby nodes:
/etc/init.d/pgService start

6. Final Service Validation
Start all remaining services on the primary node, and then on the secondary nodes if they are not already started: (Start services following section vSphere Method Graceful Shutdown and Power On of a VMware Identity Manager PostgreSQL cluster)
service vpostgres start
/etc/init.d/pgService start
service horizon-workspace start
/etc/init.d/opensearch start

7. Health Check
Log in to "https://<vIDM-FQDN>/SAAS/admin/systemDiagnostic" and verify all components show a green status without warnings.

Additional Information

Recovering vIDM PostgreSQL Cluster from Orchestration Script Mismatch and WAL Gaps due to out-of-order patching