vCenter operation times out with the error: Operation failed since another task is in progress

search cancel

vCenter operation times out with the error: Operation failed since another task is in progress

book

Article ID: 310967

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Default vCenter Server timeouts may be insufficient for lengthy operations, such as large snapshot deletions or migrations. This article provides instructions on how to adjust these settings to prevent premature timeouts.

You may encounter one or more of the following issues:

Tasks running longer than 15 minutes are forced to time out by vCenter Server, even though execution continues successfully on the ESXi host after 15 minutes.
vCenter Server tasks frequently time out or stop responding.
Backup jobs report as failed in vCenter, but the underlying task remains active on the host.
Attempts to re-run a timed-out task return the error: Operation failed since another task is in progress.
Snapshot deletions time out or fail, leading to indefinitely growing snapshots.
Lengthy warm or cold virtual machine migrations time out before completion.
Operations dependent on continuous vCenter Server connectivity—such as vMotion, Storage vMotion, or Site Recovery Manager (SRM) replication—fail.
Lengthy VMware Update Manager (VUM) tasks time out.
If email alarms are configured, you may receive alert messages such as: Alarm Host connection state on esx.example.com sent e-mail to [email protected].

Environment

VMware vSphere ESXi
VMware vCenter Server

Resolution

vCenter Server has a default 15 minute timeout for any task. For tasks that take longer than 15 minutes, you can configure the timeout values by editing the vpxd.cfg file and the vpxa.cfg file of the source and destination ESXi host.

Note:

If your backup failed due to a timeout, check your virtual machine for a backup snapshot that has been left behind.
If a backup job reports as failed in vCenter while the underlying task remains active on the host, review the pending tasks on the VM to determine the root cause of the delay. See KB article, Collecting information about tasks in VMware ESXi

Lengthy Tasks which Time Out

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted. For information on how to identify tasks within a VMware ESXi host, see Collecting information about tasks in VMware ESXi.

Note: In the case of snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. You can verify by observing the .vmdk file for the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.

For tasks that require a change temporarily eg Cross vCenter vMotion for large VMs, these settings should be reverted to default once the activity has completed

vCenter Server Timeout Settings

To change the timeout value in the vCenter Server, update vpxd.cfg on vCenter.

Log in to the vCenter Server via SSH as root user.

Note:
a) Take backup of the vpxd.cfg file using below command

cp vpxd.cfg vpxd.cfg.backup
b) Or Take Snapshot of the vCenter Server (Offline Snapshots of the respective vCenter Servers if they are in Linked Mode).
Open the vpxd.cfg file in a text editor using below command.

vi /etc/vmware-vpx/vpxd.cfg
To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file:

<config>
...
<task>
<timeout>10800</timeout>
</task>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file:

<config>
...
<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
Press Esc + : + wq! to save and exit vpxd.cfg file.
Restart the vCenter Server service using below command.
service-control --stop vmware-vpxd && service-control --start vmware-vpxd
For more information, see Stopping, Starting or Restarting appliance services.

ESXi timeout settings on builds 7u2 and earlier

Log in to the ESXi host as root via the console or an SSH session.
Open the vpxa.cfg file in a text editor using below command.

vi /etc/vmware/vpxa/vpxa.cfg
To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file:

<config>
...
<task>
<timeout>10800</timeout>
</task>
...
</config>
To increase the SOAP layer blocking call timeout, add these values in the vpxa.cfg file:

<config>
...
<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>
...
</config>

Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds or 3 hours.
Configure the timeout value for the time that vCenter Server waits to capture the virtual machine's ID at ESXi destination. Add a new configurable parameter in the vpxa.cfg file:

<config>
...
<vpxa>
...
<vmotion>
<vmIdAcquireTimeout>600</vmIdAcquireTimeout>
</vmotion>
...
</vpxa>
...
</config>

Note: The value 600 can be changed based on your requirements. This example uses 600 seconds or 10 minutes.

6. Press Esc + : + wq! to save and exit vpxa.cfg file.

7. Restart the vpxa service on the ESXi host. For more information, see Restarting the Management agents on an ESXi or ESX host.

ESXi timeout settings on builds 7u3 and later

Export the current vpxa configuration as a json file.
Make the changes needed on the json file.
Import the changed from the json file back into the config store
Restart the vpxa service.

Note: The host_ip and server_ip which contain the current ESXi management IP address and the vCenter Server IP address, are not part of the vpxa service configuration in the config store but are instead stored in the vpxa solution user configuration.

Steps to configure the settings for the vpxa service:

Note: To run the below commands, please login to the ESXi Host (CLI) through SSH using the root account. For more information, see Using ESXi Shell in ESXi 5.x, 6.x and 7.x (311213)

To display the current vpxa service configuration settings that are stored in the config store:

/bin/configstorecli config current get -c esx -g services -k vpxa
To export the current settings as a json file (the file will be exported to the same directory where the command is run from):

/bin/configstorecli config current get -c esx -g services -k vpxa -outfile tmp.json
Backup the current configuration:

cp tmp.json tmp.json.bak
Using text editor, apply the needed changes on the tmp.json file, save the changes and exit. For more information, see Editing files on an ESXi host using vi (345284):

vi /tmp.json

4.1 Edit the timeout setting change the following values
The values can be changed based on your requirements

- Change the task timeout value from to 10800

"task": {
...

   "timeout": 10800,

   },

- Change the vmacore soap session timeout value to 10800

   "vmacore": {
...

   "soap": {
   "session_timeout": 10800
   },

- Change the vmotion vm_id_acquire_timeout value to 600

   "vmotion":{

   "vm_id_acquire_timeout": 600

- Save the file

esc
:wq!
Run the command to apply the file to the database:

/bin/configstorecli config current set -c esx -g services -k vpxa -infile tmp.json

The output should look like the below:

Set: completed successfully
Restart vpxa service:

/etc/init.d/vpxa restart
Validate the current configuration settings have applied the changes

/bin/configstorecli config current get -c esx -g services -k vpxa -outfile tmp_current.json

Additional Information

Restarting the Management agents in ESXi

Stopping, Starting or Restarting appliance services

Collecting information about tasks in VMware ESXi

vSphere ESXi 7.0 U3 and later version VPXA configuration properties

Feedback

thumb_up Yes

thumb_down No