Stale tasks observed within vCenter - HCX

Products

• VMware HCX
• VMware Cloud on AWS
• VMware Cloud on Dell EMC
• VMware vSphere ESXi

Issue/Introduction

This KB article provides steps for clearing outdated HCX task from the vCenter "Recent Tasks" list.

HCX migration or deactivation tasks appear stale in the vCenter "Recent Tasks" list after completion or cancellation in the HCX UI.  This inconsistent state can prevent the successful removal of an HCX Cloud SDDC deployment.

Symptoms
• The vCenter "Recent Tasks" list shows migration tasks at 0% or stuck at a specific percentage.
• The SDDC status shows: HCX Cloud Manager on this SDDC is in failed state. Deactivate and attempt to activate again.
• The HCX UI shows migrations as completed or cancelled.
• HCX deactivation workflows fail repeatedly.

Environment

VMware Cloud on AWS

VMware HCX 4.x

VMware vSphere 7.x / 8.x

Cause

This issue occurs when vCenter fails to process the final task completion status from HCX due to temporary network contention or communication interruption between the appliances and the vCenter database.

Resolution

1. Follow the KB for clearing hung, stuck, or stale tasks from the "Recent Tasks" Panel.
   If the issue is still not resolved after following the above KB, move on to the next steps:
   
   
2. Cancel/Remove the stale task from within the vCenter MOB (Managed Object Browser):
   1. Access the Cloud vCenter MOB - https://<vcenter_FQDN>/mob/?moid=TaskManager.
   2. In the MOB page of vCenter, Click on Content , search for the TaskManager,
   3. Under RecentTask Field, click on each Task Info and ensure the task corresponds to the Task that is listed in the Recent Task.
   4. Manually inspect the info property and note down the descriptionID and startTime of the task to ensure that you are on the right task that needs to be acknowledged.
   5. Once the task is identified, invoke SetTaskState with parameters as below:
      state : success
result: leave empty
fault: leave empty
   6. Click on InvokeMethod

The above step marks the stale task as success

Workaround:

If removing the stale HCX task within the vCenter MOB is not successful, please create an SR (Creating and managing Broadcom support cases) with VMC GS with the ask to help clear the stale task from vCenter.
 
Please Note: A restart of the VPXD service may be needed and it will impact any ongoing migrations as well as access to the Cloud vCenter for about 2-3 minutes.
To restart the VPXD service:

1. Log in to the vCenter Server Appliance (VCSA) as root via SSH.
2. Restart the VPXD service:
   # service-control --restart vmware-vpxd

Additional Information

Impact/Risks:
There should be no impact to the vCenter or HCX migrations. This is just a stale task that is seen within the vCenter that can be removed by following the resolution steps and or workaround.

If the issue persists, contact Broadcom Support.