[SDDC]Synchronize domain inventory fails with error "Resource is already locked by OPERATIONS_MANAGER service"
search cancel

[SDDC]Synchronize domain inventory fails with error "Resource is already locked by OPERATIONS_MANAGER service"

book

Article ID: 407073

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • The task would fail with the following error.
Description Synchronize domain inventory
Progress Messages Import operation for sync with id ########-####-####-####-########### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ on the SDDC Manager appliance

Error

Message: Import operation for sync with id ########-####-####-####-########### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ on the SDDC Manager appliance

Remediation Message:

Reference Token: #####

Cause:

 

  • Error in var/log/vmware/vcf/domainmanager/brownfield/#####-###-###-###-#####/vcf_brownfield.log 
    [INFO] vcf_brownfield:961: Going to sync domain domain_name
[INFO] vcf_brownfield:630: Operation syncstarted on target: domain_name
[INFO] request_helper:25: Response status from retrieving domain: 200
[INFO] sddc_manager_helper:500: Using cached SDDC Manager token header
[INFO] request_helper:25: Response status from validating domain lock availability: 200
[ERROR] sync_domain:211: Could not proceed with sync operation since domain with Id: ########-####-####-####-########### is locked by an SDDC Manager operation. Errors: Lock for resource domain_name of type domain cannot be acquired by ########-####-####-####-########### service and ########-####-####-####-########### task. Reason: Resource is already locked by OPERATIONS_MANAGER service and ########-####-####-####-########### task.
[CRITICAL] vcf_brownfield:1002: Failed to sync domain domain_name
Traceback (most recent call last):
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py", line 979, in sync_domain
  sync_domain_result: list[str] = sync.sync_domain()


  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line 54, in sync_domain
   raise DomainSyncException(ErrorMessages.SYNC_LOCK_UNAVAILABLE_FOR_DOMAIN_FORMAT.format(domain_id))
common.common_exceptions.DomainSyncException: Could not proceed with sync operation since domain with Id: ########-####-####-####-########### is locked by an SDDC Manager operation. Please review the running tasks in SDDC Manager UI -> Tasks panel. Retry the sync operation once there are no running tasks.
[INFO] vcf_brownfield:633: Operation ESC[93msyncESC[00m completed on target: domain_name with status:FAIL
 
 
  • Error in var/log/vmware/vcf/domainmanager/domainmanager.log 
    DEBUG [vcf_dm,########################,b172] [c.v.v.v.s.b.u.BrownfieldImportProgressReportService,dm-####-31]  Deserialized progress report of the task with id ########-####-####-####-###########: {"
process_id":"########-####-####-####-###########","status":"completed with error","progress":{},"total_steps":4,"errors":[{"error_code":"OPERATION_ISSUE","message":"Could not proceed with sync operation since domain with Id: ########-####-####-####-########### is locked by an SDDC Manager operation. Errors: Lock for resource ############ of type domain cannot be acquired by ########-####-####-####-########### service and ########-####-####-####-########### task.
Reason: Resource is already locked by OPERATIONS_MANAGER service and ########-####-####-####-########### task.","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"Failed to sync domain domain_name","severity":"ERROR"}]}


ERROR [vcf_dm,########################,b172] [c.v.e.s.o.model.error.ErrorFactory,dm-####-31]  [#####] IMPORT_OPERATION_FAILED Import operation for sync with id ########-####-####-####-########### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name domain_name --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ on the SDDC Manager appliancecom.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Import operation for sync with id ########-####-####-####-########### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name ########### --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ --non-interactive.
Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/########-####-####-####-###########/ on the SDDC Manager appliance
 
 

Environment

VMware Cloud Foundation 9.0 

Cause

A stale exclusive lock entry existed in the SDDC Manager lock table for the domain resource, preventing the inventory sync process from acquiring the necessary lock.

Resolution

  1. Take a snapshot of the SDDC manager VM

  2. SSH into the SDDC Manager appliance as the VCF user and switch to root user.

  3. Validate if a lock exists. 

    psql -h localhost -U postgres -d platform -c "select * from lock;"


    Example Output:

    psql -h localhost -U postgres -d platform -c "select id,resource_lock_type from lock;"
                         id             | resource_lock_type
   --------------------------------------+-----------------
1#######-####-####-####-########### | EXCLUSIVE   
2#######-####-####-####-########### | LIMITED
3#######-####-####-####-########### | LIMITED
4#######-####-####-####-########### | LIMITED
            (4 rows)
  4. If there is a lock, Delete the DB lock entry with the resource_lock_type - "EXCLUSIVE": 
    psql -h localhost -U postgres -d platform -c "delete from lock where id='1#######-####-####-####-###########';"
  5.  Restart all the services in the SDDC manager by running this command 
    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
  6. Re-try the task post implementing the above steps.

Additional Information

Powered by Wolken Software