Error: "Unable to upgrade the database: Could not run cleanup for running jobs!" when upgrading VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Attempting to upgrade the VMware Cloud Director Database fails.
• When running the Database Upgrade command /opt/vmware/vcloud-director/bin/upgrade an error is returned of the form:

Examining database at URL: jdbc:postgresql://#.#.#.#:####/vcloud?socketTimeout=90&ssl=true
The next step in the upgrade process will change the VMware Cloud Director database schema.
Backup your database now using the tools provided by your database vendor.
Enter [Y] after the backup is complete. y
Unable to upgrade the database: Could not run cleanup for running jobs!

• A Delete vApp task has been running for the last 24 hrs.

Environment

VMware Cloud Director 10.x

Cause

Tasks in VMware Cloud Director can remain in a Running state if an error occurs such as if a VMware Cloud Director Cell is shut down unexpectedly.

If these running tasks remain they can block the Database Upgrade command.

Resolution

To change the status of a Running Task use the fail-tasks command of the cell management tool on one Cell in the cluster after shutting down the Cloud Director service on all Cells.
Details on how to use the fail-tasks command are outlined in the Cloud Director documentation here, Update the Failure Status of a Task .
The Cloud Director service can be started again on the Cells after running the command.
If issues persist with the upgrade after taking these steps please open a Support Request with VMware Support.

 

Additional Information

Impact/Risks:
Ensure the VMware Cloud Director service is stopped on all Cells before attempting to update the status of a failed task using the fail-tasks command of the cell management tool.
You cannot use the fail-tasks command unless all cells have been shut down.

For more information on stopping tasks in VCD see the article, Running Tasks cannot be stopped from the Cloud Director Portal.