vapp task stuck

Products

• VMware Cloud Director

Issue/Introduction

• Error when running the Database Upgrade command /opt/vmware/vcloud-director/bin/upgrade
  Unable to upgrade the database: Could not run cleanup for running jobs!
• The delete vapp task has been running for the last 24 hrs

Environment

10.x

Cause

Tasks in VMware Cloud Director can remain running if an error occurs, such as if a VMware Cloud Director Cell is shut down unexpectedly.

Resolution

1. To change the status of a Running Task, use the fail-tasks command of the cell management tool on one Cell in the cluster after shutting down the Cloud Director service on all Cells.
   Details on how to use the fail-tasks command are outlined in the Cloud Director documentation here: Update the Failure Status of a Task .
   
   
2. The Cloud Director service can be started again on the Cells after running the command.

If the fail-tasks command of the Cell Management Tool, or a restart of the Cloud Director services on all Cells, does not resolve the issue, then you can manually fail the tasks. 

1. Ensure the Cloud Director service vmware-vcd is stopped on ALL Cells.
2. Take a backup of the Cloud Director database, Backup and Restore of VMware Cloud Director Appliance.
3. SSH to Primary Cell of the affected site.
4. Connect to the Cloud Director database:
   sudo -u postgres psql vcloud;

5. Check for Tasks and Objects that remain in a Running or Busy state:
   
   SELECT * FROM jobs WHERE status IN (0,1);
   SELECT * FROM last_jobs WHERE status IN (0,1);
   SELECT * FROM task WHERE job_id IN (SELECT job_id FROM jobs WHERE status IN (0,1));
   SELECT * FROM busy_object WHERE task_id IN (SELECT id FROM task WHERE job_id IN (SELECT job_id FROM jobs WHERE status IN (0,1)));

6. Delete the busy_object and task table entries to clear the busy state of the affected objects, and set the jobs and last_jobs entries to mark the Task as failed. Use the job_id of the specific Task you wish to fail, which was found above:
   
   DELETE FROM busy_object WHERE task_id = (SELECT id FROM task WHERE job_id = 'xxx');
   DELETE FROM task WHERE job_id = 'xxx';
   UPDATE jobs SET status = 3 WHERE job_id = 'xxx';
   UPDATE last_jobs SET status = 3 WHERE job_id = 'xxx';

7. While the vmware-vcd service is stopped on all Cells and a database backup has been taken, then the complete suite of activity and scheduler tables can be cleared:
   
   TRUNCATE TABLE activity;
   TRUNCATE TABLE scheduled_activity_jobs;
   TRUNCATE TABLE activity_pc_queue;
   TRUNCATE TABLE activity_pc_event_queue;
   TRUNCATE TABLE fifo_activity_queue;
   TRUNCATE TABLE task_activity_queue;
   TRUNCATE TABLE vc_activity_queue;
   TRUNCATE TABLE activity_stats_queue;
   TRUNCATE TABLE activity_vsm_listener_queue;

8. Restart the vmware-vcd service on the Cells and confirm the affected Tasks are no longer running but have changed to a failed state.

Additional Information

Impact/Risks:
Ensure the VMware Cloud Director service is stopped on all Cells before attempting to update the status of a failed task using the fail-tasks command of the cell management tool.
You cannot use the fail-tasks command unless all cells have been shut down.