ESXi Password Remediation Fails in VCF Ops Due to Resource Lock

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Remediation of the ESXi password fails within VMware Cloud Foundation Operations (VCFOps). The targeted ESXi host displays a disconnected status in both VCF Ops and SDDC Manager.

When attempting to remediate the password from SDDC Manager, the following error message is presented:

 
Cause: Error in acquiring resource locks Lock for resource ESXI of type domain cannot be acquired by OPERATIONS_MANAGER service and #######-##95-45ee-##b4-7ce48ad#### task. Reason: Resource is already locked by OPERATIONS_MANAGER service and #####5-##6b-##f5-9aea- ######52be task

Environment

VMware Cloud Foundation Operations
VMware Cloud Foundation SDDC Manager
VMware vSphere ESXi

Cause

A stale exclusive resource lock in the SDDC Manager database and an out-of-band ESXi root password change prevents the OPERATIONS_MANAGER from synchronizing credentials, leading to account lockouts and disconnected state

Resolution

1. Log in to the SDDC Manager via SSH and check for exclusive locks in the PostgreSQL database: psql -h localhost -U postgres -d platform -c "select id,resource_lock_type from lock;"
2. Delete the identified exclusive lock: psql -h localhost -U postgres -d platform -c "delete from lock where id='<####>';"
   
   
3. Backup and remove the opasswd file on the ESXi host: cp /etc/security/opasswd /etc/security/opasswd.bak
rm -f /etc/security/opasswd
   
   
4. Reset the ESXi root password using the passwd command. Note: Set this password to the exact password currently listed in the SDDC Manager lookup_passwords table: passwd root
   
   
5. Log in to the ESXi Host UI as root. Navigate to Security and Users, select the specific service account, and change the password to a new value.( Since the Service account for the ESXI was disconnected, and service account needs to be connected first before root account of ESXI ) 
   
   
6. Return to the SDDC Manager UI and update the password for the service account to match the value set in Step 6. This will transition the service account to a connected state.
   
   
7. Remediate the root password in SDDC Manager for the ESXi host using the password present in the lookup_passwords table. This will transition the root account to a connected state.
   
   
8. Using SDDC Manager, initiate a password rotate command for service account of ESXI host

Additional Information

To check locks on SDDC, refer to
Error: Password Manager operations working on related resources - SDDC Manager

Multiple tasks in SDDC Manager fails with Error: ‘This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources.’