Certificate Task Stuck in vCenter while triggered from Fleet Manager, certificate is replaced but the task is stuck at 100%

Products

• VMware vCenter Server

Issue/Introduction

• Following a certificate replacement operation for fleet components (vCenter, ESXi, SDDC Manager) initiated via Fleet Management, a task i.e.,  "com.vmware.vcf.CERT_REPLACEMENT.label" hangs at 100% within the vCenter Server UI Recent Tasks pane.
  
  
• The initiator of the hung task in vCenter is listed as an SDDC Manager service account
• Both Fleet and SDDC tasks are completed successfully
• The task status remains stuck in a running or incomplete state indefinitely, and survives a reboot of vCenter/ SDDC/ Fleet
  
• The task itself is successful - i.e.  the underlying Certificate is actually replaced successfully on the component.
  
  
  
• There is a difference between this scenario and KB https://knowledge.broadcom.com/external/article?articleNumber=433675
• i.e. the Task name is "com.vmware.vcf.CERT_REPLACEMENT.label" and under Details it states "com.vmware.vcf.CERT_REPLACEMENT.summary"

Environment

VMware vCenter Server 9.x
VCF Operations
Fleet Manager 

Cause

The issue is caused by vCenter not passing the SDDC Manager plugin task to vCenter system properly.
Task name should not contain "com.vmware.vcf.<TBD>" instead, it should state the accurate task name in vCenter.
In this case the task name is "Certificate Operation" and under Details it should state "Certificate Operation:<sub-task>" [generate CSR/ generate certificate/ replace certificate]

As an example, a normal, successful task looks like this:

Resolution

1. Snapshot the vCenter [if Enhanced Linked Mode/ ELM in use please review https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html - it's less likely to occur, as ELM is deprecated in vSphere 9]
   
   
2. Remove the Plugin from vCenter
   Log in to the vSphere Client as an administrator.
   Navigate to Menu > Administration > Solutions > Client Plugins.
   Locate the SDDC Manager Remote Plugin.
   Select it and click Remove
   
   
   
3. Restart the SDDC Manager UI Service
   SSH into the SDDC Manager VM as the vcf user.
   Switch to the root user: su -
   Restart the UI service
   # systemctl restart sddc-manager-ui-app
This action triggers the plugin to be pushed and registered again in vCenter.
   
   
4. Verify Registration
   Log out and log back into the vSphere Client.
   Check for the SDDC Manager plugin icon in the menu

Additional Information

The task is not present in the VCDB, as a running task
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "\x" -c "select * from vpx_task;" > /tmp/vpx_task
# grep -i running /tmp/vpx_task