Old jobs may run after the vmware-vcd.service is restarted

Products

• VMware Cloud Director

Issue/Introduction

• A large number of virtual machines (Approximately 150 in total etc.) across multiple organization VDCs are being sequentially powered off/on via the API.
• The VCD service was restarted.

  # systemctl stop vmware-vcd.service
  # systemctl start vmware-vcd.service

• A running virtual machine was unexpectedly powered off by PowerOffVM_Task.
• The vcloud-container-debug.log contains output from ResumeInterruptedActivitiesJob.
  

  yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] Executing cell recovery
  yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] ** CellRecoveryJob instances found
  yyyy-mm-ddThh:mm:ss,nnn | DEBUG    | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] Running CellRecoveryJob: ResumeInterruptedActivitiesJob
  ...
  yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | ResumeInterruptedActivitiesJob | Resumed activity com.vmware.vcloud.val.internal.impl.PowerOffVmActivity
  yyyy-mm-ddThh:mm:ss,nnn | DEBUG    | vim-proxy-activity-pool-* | PowerOffVmActivity             | [Activity Execution] Running phase: PowerOffVmPhase - Handle: urn:uuid:[***], Current Phase: PowerOffVmActivity$PowerOffVmPhase | requestId=[***],request=POST https://[***]/action/undeploy,requestTime=[***]

• When converting the Unix timestamp in milliseconds found in requestTime, it results in a date/time in the past. 
  Below is an example command to check the date/time for a requestTime value of 1743172000000.

  $ date -d "@$((1743172000000 / 1000))" +"%Y/%m/%d %T"
  2025/03/28 23:26:40

Environment

VMWare Cloud Director 10.6.*

Cause

This is because the completed_at field in the activity table of the VCD database remains NULL, indicating that the activity has not been completed.

Resolution

This issue has been resolved in VCD 10.6.2. Refer to the Resolved issues in the VMware Cloud Director 10.6.2 Release Notes. Download VMware Cloud Director 10.6.2 from Broadcom Support Portal.
To resolve this issue after upgrading to VCD 10.6.2, the feature flag vcloud.activities.resume.forcefullyMarkExpiredActivitiesComplete must be set to true. By default, this feature flag is set to false.

To enable vcloud.activities.resume.forcefullyMarkExpiredActivitiesComplete, run the following command on any VCD cell:
$VCLOUD_HOME/bin/cell-management-tool manage-config -n vcloud.activities.resume.forcefullyMarkExpiredActivitiesComplete -v true

The setting can be verified with the following command:
$VCLOUD_HOME/bin/cell-management-tool manage-config -n vcloud.activities.resume.forcefullyMarkExpiredActivitiesComplete -l

Workaround:
For versions prior to VCD 10.6.2, contact Broadcom Support and reference this KB article in the case description. For more information, see "Creating and managing Broadcom cases".