VCF SDDC Manager shows multiple stuck tasks with "Fetching task/subtask info" due to Deployment Lock

Products

• VMware SDDC Manager

Issue/Introduction

• In the SDDC Manager UI, the Tasks panel displays multiple entries with the status "Fetching subtask info..." or "Fetching task info..." which are stuck and persist indefinitely.

• Restarting services via /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh command or rebooting the SDDC Manager does not clear them nor bring any change to the behaviour.

• Reviewing the Tasks list show there is a SDDC-Manager Backup Task in stuck state without any progress which cannot be cancelled.

• Execution of the SOS command to collect support logs on SDDC Manager fails returning an error referring to deployment lock:

Found active workflow for <id>, Description: SDDC Manager DEPLOYMENT lock to perform Backup & Restore operation.

Example:

• Querying the SDDC Manager database using the below command further confirms an active deployment lock:

/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "select * from lock;"

The output of the above will depict an active lock with resource_type as deployment as observed on below example:

Cause

This issue is caused by a stale lock held by a stuck SDDC Manager Backup operation typically caused by an interrupted or timed-out SDDC Manager backup workflow. Because this lock persists in the database, it may not get cleared by restarting SDDC Manager services or rebooting the SDDC Manager appliance.

Resolution

To resolve this issue, take a snapshot of the SDDC-Manager and perform the manual cleanup steps shared below:

1. Log in to the SDDC Manager via SSH as vcf and switch to root

2. Check the current status of the lock: /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "select * from lock;"

3. Delete the lock: /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

Example: