Old jobs may run after the vmware-vcd.service is restarted.
search cancel

Old jobs may run after the vmware-vcd.service is restarted.

book

Article ID: 406271

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

  • A large number of virtual machines (Approximately 150 in total etc.) across multiple organization VDCs are being sequentially powered off/on via the API.
  • The VCD service was restarted. 
    # systemctl stop vmware-vcd.service
    # systemctl start vmware-vcd.service
  • A running virtual machine was unexpectedly powered off by PowerOffVM_Task.
  • The vcloud-container-debug.log contains output from ResumeInterruptedActivitiesJob.
    yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] Executing cell recovery
    yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] ** CellRecoveryJob instances found
    yyyy-mm-ddThh:mm:ss,nnn | DEBUG    | Backend-activity-pool-**  | LockRecoveryJob                | [Activity Execution] Running CellRecoveryJob: ResumeInterruptedActivitiesJob
    ...
    yyyy-mm-ddThh:mm:ss,nnn | INFO     | Backend-activity-pool-**  | ResumeInterruptedActivitiesJob | Resumed activity com.vmware.vcloud.val.internal.impl.PowerOffVmActivity
    yyyy-mm-ddThh:mm:ss,nnn | DEBUG    | vim-proxy-activity-pool-* | PowerOffVmActivity             | [Activity Execution] Running phase: PowerOffVmPhase - Handle: urn:uuid:[***], Current Phase: PowerOffVmActivity$PowerOffVmPhase | requestId=[***],request=POST https://[***]/action/undeploy,requestTime=[***]
  • When converting the Unix timestamp in milliseconds found in requestTime, it results in a date/time in the past. 
    Below is an example command to check the date/time for a requestTime value of 1743172000000. 
    $ date -d "@$((1743172000000 / 1000))" +"%Y/%m/%d %T"
    2025/03/28 23:26:40

Environment

VMWare Cloud Director 10.6.*

Cause

This is because the completed_at field in the activity table of the VCD database remains NULL, indicating that the activity has not been completed.

Resolution

Broadcom is aware of the issue and a fix is planned for a future release of VCD.
As a workaround, to perform the modifications to the VCD DB prior to restarting the VCD service.

Workaround:

Option 1: Delete activity table by refer to fifo_activity_queue table.
[Procedure]
0. Take a backup of the VCD database in advance
Please proceed only after ensuring that the system can be fully restored.

1. Perform DELETE on the primary cell
1-1. Connect to the VCD database

sudo -i -u postgres psql vcloud

1-2. Check before deleting

SELECT id,activity_name,phase_name,state_handle,is_cancelable,is_cancel_requested,is_supervised_req,is_supervised_req_complete,is_supervisor_timedout,framework_token,entity_id,completed_at,submitted_by,running_on,result,exception,partial_failure,hash_id
FROM activity
WHERE state_handle NOT IN (
    SELECT activity_state_handle FROM fifo_activity_queue)
AND completed_at IS NULL 
AND activity_name NOT IN (
    'com.vmware.vcloud.common.activity.scheduled.JobSchedulingActivity',
    'com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader',
    'com.vmware.vcloud.backendbase.management.system.TaskActivity',
    'com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity',
    'com.vmware.vcloud.metrics.statsfeeder.StatsFeederCollectorActivity');

1-3. Execute DELETE

DELETE FROM activity 
WHERE state_handle NOT IN (
      SELECT activity_state_handle FROM fifo_activity_queue)
AND completed_at IS NULL 
AND activity_name NOT IN (
    'com.vmware.vcloud.common.activity.scheduled.JobSchedulingActivity',
    'com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader',
    'com.vmware.vcloud.backendbase.management.system.TaskActivity',
    'com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity',
    'com.vmware.vcloud.metrics.statsfeeder.StatsFeederCollectorActivity');

1-4. Disconnect from the VCD database

\q


Option 2: Truncating the activity-related tables.
[Procedure]
0. Take a backup of the VCD database in advance
Please proceed only after ensuring that the system can be fully restored.

1. Set Failover Mode to 'manual'

2. Quiesce and put all VCD cells into maintenance mode, then stop the services
(Do this in the order: application cell -> standby cell -> primary cell)

$VCLOUD_HOME/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -q true
$VCLOUD_HOME/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -maintenance true
$VCLOUD_HOME/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s

3. Perform TRUNCATE on the primary cell
3-1. Connect to the VCD database

sudo -i -u postgres psql vcloud

3-2. Execute TRUNCATE

TRUNCATE TABLE activity;
TRUNCATE TABLE scheduled_activity_jobs;
TRUNCATE TABLE activity_pc_queue;
TRUNCATE TABLE activity_pc_event_queue;
TRUNCATE TABLE fifo_activity_queue;
TRUNCATE TABLE task_activity_queue;
TRUNCATE TABLE vc_activity_queue;
TRUNCATE TABLE activity_stats_queue;
TRUNCATE TABLE activity_vsm_listener_queue;

3-3. Disconnect from the VCD database

\q

4. Start services on all VCD cells (in the order: primary cell -> standby cell -> application cell)

systemctl start vmware-vcd

Confirm that the cells are not in maintenance mode or quiesced

$VCLOUD_HOME/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) --status

Additional Information

If this issue occurs, consider to contact VMware by Broadcom Support and note this KB in the problem description.
For more information, see "Creating and managing Broadcom support request (SR) cases".

Powered by Wolken Software