VMware Identity Manager Patch Upgrade Hangs During GRUB2 Pre-Checks Due to Concurrent Execution

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

During the patch installion of the VMware Identity Manager (vIDM) patch (CSP-102092), the installation script stalls indefinitely. This hang occurs specifically during the execution of environment pre-checks, immediately after successfully detecting and validating the presence of /boot/grub2/grub.cfg.

The patch log execution halts at the following state:

YYYY-MM-DD  - Disk space at /boot: 78MB free 
YYYY-MM-DD - All checks passed for ZIP '/db/CSP-102092-Appliance-3.3.7-Patch/CSP-102092-Appliance-3.3.7.zip'. 
YYYY-MM-DD - Running on node: <REDACTED_HOSTNAMES> 
YYYY-MM-DD - Checking grub2 presence 
YYYY-MM-DD - grub2 detected: /boot/grub2/grub.cfg exists 

Inspecting active environment processes with ps -ef | grep CSP-102092 reveals duplicate instances of the patch automation script running concurrently on the node:

root     15305 31717  0 04:11 pts/0    00:00:00 /bin/bash ./CSP-102092-patch-automation.sh -f CSP-102092-Appliance-3.3.7.zip -r
root     15370 15305  0 04:11 pts/0    00:00:00 /bin/bash ./CSP-102092-patch-automation.sh -f CSP-102092-Appliance-3.3.7.zip -r

To validate the lock contention state on the hanging process, query the wait channel and open file descriptors for the identified PID:

cat /proc/<PID>/wchan
ls -l /proc/<PID>/fd

The resulting output will confirm the child processes are in an unrecoverable deadlock waiting indefinitely for the release of the RPM lock:

futex_wait_queue_me
...
lr-x------ 1 root root 64 Jun  3 04:15 3 -> /var/lib/rpm/.rpm.lock

Environment

VMware Identity Manager 3.3.7

 

Cause

The root cause is the concurrent execution of multiple instances of the patch deployment script (CSP-102092-patch-automation.sh). This overlapping execution creates a race condition over system resources, leading to database lock contentions within the underlying package manager (rpmdb). This deadlocks the child processes, causing them to wait indefinitely for the release of /var/lib/rpm/.rpm.lock and local runtime database files.

Resolution

To clear the lock contention and successfully complete the patch installation, follow the steps below sequentially:

Step 1: Forcefully Terminate Stale Patch Processes

Verify the unrecoverable deadlock state and kill all hanging script processes across the subsystem:

kill -9 $(pgrep -f CSP-102092-patch-automation.sh)

Step 2: Clear Stale Package Manager Locks

Manually remove the locked RPM database files to clear the resource contention:

rm -f /var/lib/rpm/.rpm.lock
rm -f /var/lib/rpm/__db*

Step 3: Rebuild the RPM Database

Re-index and reconstruct the RPM database layout to clear potential database metadata corruption:

rpm --rebuilddb

Step 4: Re-initiate the Patch Deployment

Restart the patch automation sequence from your extracted patch source directory:

./CSP-102092-patch-automation.sh -f CSP-102092-Appliance-3.3.7.zip -r