[SDDC]Synchronize domain inventory fails with error "Resource is already locked by OPERATIONS_MANAGER service"
search cancel

[SDDC]Synchronize domain inventory fails with error "Resource is already locked by OPERATIONS_MANAGER service"

book

Article ID: 407073

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

The task would fail with the following error.

Description Synchronize domain inventory
Progress Messages Import operation for sync with id 79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx/ on the SDDC Manager appliance

Error

Message: Import operation for sync with id 79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/79b288cd-0cc0-482a-b5c5-xxxxxxxxxxxxxx/ on the SDDC Manager appliance

Remediation Message:

Reference Token: AAVJNS

Cause:

 

In the logs, you will find these snippets

var/log/vmware/vcf/domainmanager/brownfield/9c6ec7c8-0fdb-4b8a-b939-xxxxxx/vcf_brownfield.log

[YYYY-MM-DD 07:22:17,467] [INFO] vcf_brownfield:961: Going to sync domain domain_name
[YYYY-MM-DD 07:22:17,467] [INFO] vcf_brownfield:630: Operation syncstarted on target: domain_name
[YYYY-MM-DD 07:22:17,591] [INFO] request_helper:25: Response status from retrieving domain: 200
[YYYY-MM-DD 07:22:17,592] [INFO] sddc_manager_helper:500: Using cached SDDC Manager token header
[YYYY-MM-DD 07:22:17,601] [INFO] request_helper:25: Response status from validating domain lock availability: 200
[YYYY-MM-DD 07:22:17,602] [ERROR] sync_domain:211: Could not proceed with sync operation since domain with Id: 0746b1b0-56eb-48e2-b218-xxxxxxxxxxxxx is locked by an SDDC Manager operation. Errors: Lock for resource domain_name of type domain cannot be acquired by 5c933940-a66b-4ea8-91fa-xxxxxxxxxxxxx service and b59122c2-551b-4cca-9838-xxxxxxxxxxxxxx task. Reason: Resource is already locked by OPERATIONS_MANAGER service and ed9bfbab-9fe4-4708-97db-018e07fc2477 task.
[YYYY-MM-DD 07:22:17,602] [CRITICAL] vcf_brownfield:1002: Failed to sync domain domain_name
Traceback (most recent call last):
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py", line 979, in sync_domain
    sync_domain_result: list[str] = sync.sync_domain()
                                    ^^^^^^^^^^^^^^^^^^
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line 54, in sync_domain
    raise DomainSyncException(ErrorMessages.SYNC_LOCK_UNAVAILABLE_FOR_DOMAIN_FORMAT.format(domain_id))
common.common_exceptions.DomainSyncException: Could not proceed with sync operation since domain with Id: 0746b1b0-56eb-48e2-b218-xxxxxxxxxxxxx is locked by an SDDC Manager operation. Please review the running tasks in SDDC Manager UI -> Tasks panel. Retry the sync operation once there are no running tasks.
[YYYY-MM-DD 07:22:17,610] [INFO] vcf_brownfield:633: Operation ESC[93msyncESC[00m completed on target: domain_name with status:FAIL
 
var/log/vmware/vcf/domainmanager/domainmanager.log 
 
YYYY-MM-DDT07:22:17.705+0000 DEBUG [vcf_dm,68999a2717510cada549fe6a94017e50,b172] [c.v.v.v.s.b.u.BrownfieldImportProgressReportService,dm-exec-31]  Deserialized progress report of the task with id 9c6ec7c8-0fdb-4b8a-b939-816e41a33a0c: {"
process_id":"9c6ec7c8-0fdb-4b8a-b939-816e41a33a0c","status":"completed with error","progress":{},"total_steps":4,"errors":[{"error_code":"OPERATION_ISSUE","message":"Could not proceed with sync operation since domain with Id: 0746b1b0-56eb-48e2-b218-95ecf2a0e4da is locked by an SDDC Manager operation. Errors: Lock for resource tdemo-m01 of type domain cannot be acquired by 5c933940-a66b-4ea8-91fa-xxxxxxxxxxxx service and b59122c2-551b-4cca-9838-xxxxxxxxxxxxtask.
Reason: Resource is already locked by OPERATIONS_MANAGER service and ed9bfbab-9fe4-4708-97db-018e07fc2477 task.","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"Failed to sync domain domain_name","severity":"ERROR"}]}
 
YYYY-MM-DDT07:22:17.715+0000 ERROR [vcf_dm,68999a2717510cada549fe6a94017e50,b172] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-31]  [9QIO59] IMPORT_OPERATION_FAILED Import operation for sync with id 9c6ec7c8-0fdb-4b8a-b939-xxxxxxxxx failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/9c6ec7c8-0fdb-4b8a-b939-xxxxxxxxx/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/9c6ec7c8-0fdb-4b8a-b939-xxxxxxx/ on the SDDC Manager appliancecom.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Import operation for sync with id 9c6ec7c8-0fdb-4b8a-b939-816e41a33a0c failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name tdemo-m01 --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/9c6ec7c8-0fdb-4b8a-b939-xxxxxxxxx/ --non-interactive.
Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/9c6ec7c8-0fdb-4b8a-b939-xxxxxxxxxx/ on the SDDC Manager appliance

Environment

VMware Cloud Foundation 9.0 

Cause

A stale exclusive lock entry existed in the SDDC Manager lock table for the domain resource, preventing the inventory sync process from acquiring the necessary lock.

Resolution

  1. Take a snapshot of the SDDC manager

  2. SSH into the SDDC Manager appliance as the root user.

  3. Validate that a lock exists. 

    psql -h localhost -U postgres -d platform -c "select * from lock;"

         or

         psql -h localhost -U postgres -d platform -c "select id,resource_lock_type from lock;"

     Example 

      psql -h localhost -U postgres -d platform -c "select id,resource_lock_type from lock;"
                     id                  | resource_lock_type
   --------------------------------------+--------------------
            cb106a85-a327-4795-92e5-XXX| EXCLUSIVE   <----
            297751f1-41e2-455a-b01e-XXX| LIMITED
            7098af18-57b6-48da-ae27-XXX| LIMITED
            99d56c1b-d857-40f9-a6b0-XXX| LIMITED
            (4 rows)

     4. Delete the DB lock entry:

    psql -h localhost -U postgres -d platform -c "delete from lock where id='lock_id from step 2';"

      Example

      psql -h localhost -U postgres -d platform -c "delete from lock where id='cb106a85-a327-4795-92e5-XXX';"

     5. Restart all the services in the SDDC manager by running this command

    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

After performing these steps, domain-related operations in SDDC Manager should proceed without issue.

Additional Information

Powered by Wolken Software