vCenter operation times out with the error: Operation failed since another task is in progress

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Default vCenter Server timeouts may be insufficient for lengthy operations, such as large snapshot deletions or migrations. This article provides instructions on how to adjust these settings to prevent premature timeouts.

You may encounter one or more of the following issues:

Tasks running longer than 15 minutes are forced to time out by vCenter Server, even though execution continues successfully on the ESXi host after 15 minutes.
vCenter Server tasks frequently time out or stop responding.
Backup jobs report as failed in vCenter, but the underlying task remains active on the host.
Attempts to re-run a timed-out task return the error: Operation failed since another task is in progress.
Snapshot deletions time out or fail, leading to indefinitely growing snapshots.
Lengthy warm or cold virtual machine migrations time out before completion.
Operations dependent on continuous vCenter Server connectivity—such as vMotion, Storage vMotion, or Site Recovery Manager (SRM) replication—fail.
Lengthy VMware Update Manager (VUM) tasks time out.
If email alarms are configured, you may receive alert messages such as: Alarm Host connection state on esx.example.com sent e-mail to admin@example.com.

Environment

VMware vSphere ESXi
VMware vCenter Server

Resolution

vCenter Server has a default 15 minute timeout for any task. For tasks that take longer than 15 minutes, you can configure the timeout values by editing the vpxd.cfg file and the vpxa.cfg file of the source and destination ESXi host.

Note:

If your backup failed due to a timeout, check your virtual machine for a backup snapshot that has been left behind.
If a backup job reports as failed in vCenter while the underlying task remains active on the host, review the pending tasks on the VM to determine the root cause of the delay. See KB article, Collecting information about tasks in VMware ESXi

Lengthy Tasks which Time Out

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted. For information on how to identify tasks within a VMware ESXi host, see Collecting information about tasks in VMware ESXi.

Note: In the case of snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. You can verify by observing the .vmdk file for the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.

vCenter Server Timeout Settings

To change the timeout value in the vCenter Server, update vpxd.cfg on vCenter.

Log in to the vCenter Server via SSH as root user.

Note:
a) Take backup of the vpxd.cfg file using below command
cp vpxd.cfg vpxd.cfg.backup
b) Or Take Snapshot of the vCenter Server (Offline Snapshots of the respective vCenter Servers if they are in Linked Mode).
Open the vpxd.cfg file in a text editor using below command.

vi /etc/vmware-vpx/vpxd.cfg
To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file:

<config>
...
<task>
<timeout>10800</timeout>
</task>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file:

<config>
...
<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
Press Esc + : + wq! to save and exit vpxd.cfg file.
Restart the vCenter Server service using below command.
service-control --stop vmware-vpxd && service-control --start vmware-vpxd
For more information, see Stopping, Starting or Restarting appliance services.

ESXi timeout settings

Log in to the ESXi host as root via the console or an SSH session.
Open the vpxa.cfg file in a text editor using below command.

vi /etc/vmware/vpxa/vpxa.cfg
To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file:

<config>
...
<task>
<timeout>10800</timeout>
</task>
...
</config>
To increase the SOAP layer blocking call timeout, add these values in the vpxa.cfg file:

<config>
...
<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds or 3 hours.
Configure the timeout value for the time that vCenter Server waits to capture the virtual machine's ID at ESXi destination. Add a new configurable parameter in the vpxa.cfg file:

<config>
...
<vpxa>
...
<vmotion>
<vmIdAcquireTimeout>600</vmIdAcquireTimeout>
</vmotion>
...
</vpxa>
...
</config>

Note: The value 600 can be changed based on your requirements. This example uses 600 seconds or 10 minutes.