Multiple tasks in SDDC Manager fails with Error: "This operation is not allowed because CERTIFICATE_OPERATION operation(s) are working on related resources."
search cancel

Multiple tasks in SDDC Manager fails with Error: "This operation is not allowed because CERTIFICATE_OPERATION operation(s) are working on related resources."

book

Article ID: 428775

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • A CSR generation operation was attempted; the task succeeded, but in the SDDC tasks UI, it shows as in progress.
  • Upon performing any further task, the SDDC manager failed with the error."This operation is not allowed because CERTIFICATE_OPERATION operation(s) are working on related resources."
  • in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log, We see the following error:
yyyy-mm-ddThh:mm:02.540+0000 INFO  [vcf_om,xxxxxxxxxxxxxxxxxxx] [c.v.v.f.t.a.FunctionalityToggleAdapterImpl,http-nio-127.0.0.1-7300-exec-1] isFunctionalitySupported: {"supported":false,"error":"This operation is not allowed because CERTIFICATE_OPERATION operation(s) are working on related resources."}
yyyy-mm-ddThh:mm:02.542+0000 ERROR [vcf_om,xxxxxxxxxxxxxxxxxxx] [c.v.v.f.t.i.FunctionalityToggleApiFilter,http-nio-127.0.0.1-7300-exec-1] Functionality blocked:
com.vmware.vcf.functionality.toggle.error.FunctionalityToggleException: The attempted functionality REPLACE_RESOURCE_CERTIFICATES is not supported in the current system state. This operation is not allowed because CERTIFICATE_OPERATION operation(s) are working on related resources.
  • Upon checking the SDDC DB, we notice 
root@sddc []# psql -h localhost -U postgres -d platform -c "select * from lock;"
                  id                  | creation_time | modification_time | error |                                                                         locking_context
                                                       | resource_id | resource_type | status | resource_name | reentrant_key | resource_lock_type
--------------------------------------+---------------+-------------------+-------+----------------------------------------------------------------------------------------------------------
-------------------------------------------------------+-------------+---------------+--------+---------------+---------------+--------------------
 aaaaaaa-bbbb-ccccccc-dddddd | 1770261009086 |     1770261009086 |       | {"serviceIdentifier":"CERTIFICATE_OPERATION","description":"Deployment lock held by certificate replaceme
nt operation.","pollingInterval":0,"expirationTime":0} |             | deployment    | ACTIVE | LEGACY        |               |
(1 row)


root@sddc []# psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, operation_status, start_time from certificatemanagement.certificate_operation where operation_status='FAILED';"
             workflow_id              | operation_type | operation_status |         start_time
--------------------------------------+----------------+------------------+----------------------------
 bb4ea150-aaaa-bbbb-cccc-ddddddddddd | GENERATE_CSR   | FAILED           | 2026-02-05 03:10:09.088605
(1 row)

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

 

Cause

In VCF, SDDC Manager uses a locking mechanism to ensure data consistency during sensitive operations like certificate replacement. If a workflow—such as the one GENERATE_CSR identified here—encounters an unhandled failure, it may not trigger the routine to release the deployment lock. Until this entry is manually removed from the lock table and the associated services are refreshed, the system assumes an operation is still in progress and blocks subsequent requests.

Resolution

  • Take a non-memory snapshot of the SDDC machine.
  • Update the workflow_id  to be successful in the operationsmanager DB.
  • psql -U postgres -h localhost -d operationsmanager -c "UPDATE certificatemanagement.certificate_operation SET operation_status='SUCCESSFUL' WHERE workflow_id='ID';"

Example: psql -U postgres -h localhost -d operationsmanager -c "UPDATE certificatemanagement.certificate_operation SET operation_status='SUCCESSFUL' WHERE workflow_id='bb4ea150-aaaa-bbbb-cccc-ddddddddddd';"

  • Deleted the DB lock entry from the platform DB

psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

  • Restart LCM service by running the command:
    systemctl restart lcm
  • Validate the SDDC Ui and check if you are now able to perform the operations.

Additional Information

Sample output of the command psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, operation_status, start_time from certificatemanagement.certificate_operation;"

 psql -U postgres -h localhost -d operationsmanager -c "select workflow_id, operation_type, operation_status, start_time from certificatemanagement.certificate_operation;"
             workflow_id              |    operation_type    | operation_status |         start_time
--------------------------------------+----------------------+------------------+----------------------------
 d8a6c43b-3af0-4340-8083-xxxxxxxx| GENERATE_CSR         | SUCCESSFUL       | 2025-01-08 18:46:08.50638
 60f54e8c-70f9-4ecd-94d4-xxxxxxxx| REPLACE_CERTIFICATE  | SUCCESSFUL       | 2025-01-08 18:47:10.582655
 29f6c6d4-71a9-4b0b-b38b-xxxxxxx| GENERATE_CSR         | SUCCESSFUL       | 2025-03-07 14:59:04.24089
 b464cbc8-0c39-471a-95fd-xxxxxxxx| GENERATE_CERTIFICATE | SUCCESSFUL       | 2025-03-07 14:59:43.844878
 1552749c-aae4-4085-acbd-xxxxxxxx| REPLACE_CERTIFICATE  | SUCCESSFUL       | 2025-03-07 15:03:33.95389
(5 rows)

 

Powered by Wolken Software