NSXT upgrade failed with task returing as cancelled

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

An NSX-T upgrade within VMware Cloud Foundation (VCF) fails and the corresponding upgrade task is marked as cancelled.

 

The folllowng erros can be found for the upgrade task ID in /var/log/vmware/vcf/lcm/lcm-debug.log

Error acquiring resource lock.

Error in acquiring resource locks Lock for resource LEGACY of type deployment cannot be acquired by LCM service and TASK_ID task. Reason: Resource is already locked by FSM service and null task ..

Environment

VMware Cloud Foundation 9.1

Cause

A stale lock entry existed in the SDDC Manager lock table for the domain resource, preventing the upgrade from acquiring the necessary lock.

Resolution

1. Take a snapshot of the SDDC manager VM

2. SSH into the SDDC Manager appliance as the VCF user and switch to root user.

3. Validate if a lock exists. 

   psql -h localhost -U postgres -d platform -c "select * from lock;"

   
   If there is a lock, Delete the DB lock entry 

4. psql -h localhost -U postgres -d platform -c "delete from lock where id='1#######-####-####-####-###########';"

5.  Restart all the services in the SDDC manager by running this command

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

6. Re-try the task post implementing the above steps.