SvMotion and vMotion jobs sitting in queued state for extended period of time

Products

• VMware vCenter Server

Issue/Introduction

Jobs sit in queued state for extended periods of time and persist in that state post vCenter reboot

Environment

• vCenter 7
• vCenter 8

Cause

Jobs aren't going into a 'failed' state and not being cleaned up as part of vCenter's periodic job clean up process.

 

This can be verified by performing the following:

Check the VPXD start time:
# head -1 /var/log/vmware/vpxd/vpxd.log
YYYY-MM-DDTHH:MM:SS.###Z INFO vpxd 3022485 [######] - time the service was last started YYYY-MM-DDTHH:MM:SS.###Z, Section for VMware VirtualCenter, pid=########, version=9.1.0, build=88414925, option=DEBUG

Now check the pending tasks against the VCDB:
# psql -U postgres VCDB -c "select task_id, queue_time, complete_state from vpx_task where complete_state = 'queued' and queue_time < 'YYYY-MM-DDTHH:MM:SS.###Z'"
task_id |       queue_time        | complete_state 
---------+-------------------------+----------------
11 | YYYY-MM-DD HH:MM:SS.### | queued
(1 row)

If the task is older than the most recent "time the service was last started" then the queued task is hung and the script can be ran.

Resolution

1. Ensure that there are snapshots of the vCenter (if in linked mode you will need to make offline snapshots of all vCenters participating in ELM).
2. Download the "Clean_Queued_Tasks.sh" script attached to this KB
3. Copy the "Clean_Queued_Tasks.sh" script to the /root directory on the vCenter
4. SSH into the vCenter
5. cd /root
6. chmod +x Clean_Queued_Tasks.sh
7.  Run the script with ./Clean_Queued_Tasks.sh
8. Check the tasks in vCenter and they should now show as "Failed"
9. In 30 minutes to an hour vCenter's clean up process should clean up the failed tasks