Snapshot creation and deletion process queued due to which backups process is failing.
search cancel

Snapshot creation and deletion process queued due to which backups process is failing.

book

Article ID: 377741

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0

Issue/Introduction

The 3rd party backup is failing from the vCenter Server side since there is a snapshot process seen to be queued on the vCenter Server, the tasks are with respect to create snapshot and remove snapshot.
Basically the queued tasks are stales in the VCDB that are not even running on the ESXi host.

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

The two tasks with respect to snapshot creation and deletion is queued on the vCenter Server with no progress and while checking for task running on ESXi host with respect to the VMs from the CLI we do not see any snapshot create or delete task.

Resolution

Note: This procedure modifies the database. Ensure to take a backup of the vCenter Server before proceeding.

  • Take snapshot of VC, if VC is in ELM make sure to take offline snapshot of all VCs.
  • SSH to vCenter Server using root credentials.
  • Please reach out Broadcom support if assistance needed while preforming the steps.

Section A

1. Check for VM snapshot task: Ensure that no VM snapshot task is in progress.
2. Identify the ESXi host: Determine the respective ESXi host where the VM is running.
3. Use the command: Identify the snapshot task on the ESXi host using the 'vim-cmd vimsvc/task_list' command.
4. Avoid using commands: Under no circumstances should the commands mentioned in Section B be used if a VM snapshot task is in progress.
5. If the output is similar to the following seen below please do not proceed forward with the steps mentioned below in section B:

        (ManagedObjectReference) [
                  'vim.Task:haTask-xxx-vim.VirtualMachine.createSnapshot-xxxx'

Section B

1 ) Take snapshot of the vCenter Server and run the following command to connect to the VCDB :

  • root@vCenter [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

2 ) Stop the vpxd service using the command :

  • root@vCenter [ ~ ]# service-control --stop vpxd

 3 ) To check the queued tasks from VCDB:  

  • VCDB=# select * from vpx_task where complete_state NOT IN('success','error') AND name IN ('vim.VirtualMachine.createSnapshot','vim.vm.Snapshot.remove');

 4 ) To delete the queued tasks from VCDB :

  • VCDB=# delete from vpx_task where complete_state NOT IN('success','error') AND name IN ('vim.VirtualMachine.createSnapshot','vim.vm.Snapshot.remove');

5 ) Exit the VCDB console :

  • VCDB=# \q

 6 ) To restart the vpxd service :

  • root@vCenter [ ~ ]# vmon-cli -r vpxd

7 )  The queued task will no longer be running, when we check from the vSphere Client as they were removed from the VCDB.

Additional Information

Powered by Wolken Software