Certificate installation via SDDC Manager fails with Error: REPLACE_RESOURCE_CERTIFICATES is not supported

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• When attempting to install or replace SSL certificates for components (vCenter, NSX, etc.) via the SDDC Manager UI, the operation fails.
• The UI displays a generic error as follows:
  
  
  
  
• Certificate replacement tasks appear stuck or fail immediately.
• But the underlying system reports that the state is invalid for the requested operation.
• From the SDDC UI, there are too many Stale Tasks are in the state of Pending/Failed/Running as follows:
  
  
  
  
  
  
  
  
• Under SDDC Manager's, /var/log/vmware/vcf/operationsmanager/operationsmanager.log, below entries are available, indicating "leftover" or "stale" tasks:
  

  YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,##########,####] [c.v.v.t.services.TaskPublisher,om-scheduler-1] Found leftover task {"_type":"Task","_value":"task-######","_serverGuid":"########-####-####-####-########"} and marked it as failed!
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,##########,####] [c.v.v.t.services.TaskPublisher,om-scheduler-1] Found leftover task {"_type":"Task","_value":"task-######","_serverGuid":"########-####-####-####-########"} and marked it as failed!

  YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,##########,####] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,http-nio-127.0.0.1-####-exec-#] Creating task for the certificate operation: VIEW_CERTIFICATE
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,##########,####] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,http-nio-127.0.0.1-####-exec-#] Creating task for the certificate operation: VIEW_CERTIFICATE

          at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)

Environment

• VCF 9.x
• SDDC Manager 5.x / 9.x

Cause

• The issue is caused by stale or orphaned tasks residing in the SDDC Manager’s internal database.
• While the system may not be executing any active processes, the SDDC Manager database may incorrectly report tasks as 'Pending' or 'Running' within the Task Status view.

Resolution

To resolve this issue, follow below steps:

1. Take a Snapshot of the SDDC Manager Appliance.
2. Clean up the Stale tasks from the SDDC Manager. For more information, refer: Cleanup stale tasks after SDDC Manager recovery
3. Restart the SDDC Manager services, using below command:
   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
   
   
4. Navigate back to the SDDC Manager UI and retry the Certificate Replacement/Installation for the required component.

Additional Information

When the operations-manager service detects previous tasks that did not close gracefully (due to a service restart, timeout, or previous failure), it enters a protective state. Because the system believes a certificate-related operation is already "in flight" or the scheduler is cluttered with "leftover" tasks, it blocks new REPLACE_RESOURCE_CERTIFICATES requests to prevent database corruption or inconsistent component states.