vCenter operation times out with the error: Operation failed since another task is in progress

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Default vCenter Server timeouts may be insufficient for lengthy operations, such as large snapshot deletions or migrations. This article provides instructions on how to adjust these settings to prevent premature timeouts.

One or more of the following issues might be encountered:

Tasks running longer than 15 minutes are forced to time out by vCenter Server, even though execution continues successfully on the ESXi host after 15 minutes.
vCenter Server tasks frequently time out or stop responding.
Backup jobs report as failed in vCenter, but the underlying task remains active on the host.
Attempts to re-run a timed-out task return errors like:
```
Operation failed since another task is in progress.
```
Snapshot deletions time out or fail, leading to indefinitely growing snapshots.
Lengthy warm or cold virtual machine migrations time out before completion.
Operations dependent on continuous vCenter Server connectivity—such as vMotion, Storage vMotion, or Site Recovery Manager (SRM) replication—fail.
Lengthy VMware Update Manager (VUM) tasks time out.

If email alarms are configured, you may receive alert messages such as:

Alarm Host connection state on esx.example.com sent e-mail to [email protected].

Environment

VMware vSphere ESXi
VMware vCenter Server
VCF 9.x

Cause

vCenter Server has a default 15 minute timeout for any task.
For tasks that take longer than 15 minutes, other timeout values can be configured by editing the vpxd.cfg file in vCenter Server appliance, as well as the vpxa.cfg (in ESXi 7.0 Update 2 and older versions) or the vpxa settings in the ESXI configuration store (for ESXi versions from 7.0 Update 3 on) of the source and destination ESXi hosts.

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted. For information on how to identify tasks within a VMware ESXi host, see Collecting information about tasks in VMware ESXi.

Resolution

Notes:
If a backup failed due to a timeout, check the virtual machine for a backup snapshot that has been left behind.
If a backup job reports as failed in vCenter while the underlying task remains active on the host, review the pending tasks on the VM to determine the root cause of the delay. See Collecting information about tasks in VMware ESXi for more information.
In case of a snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. This can be verified by observing the .vmdk file of the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.
For tasks that require a change temporarily (e.g. Cross vCenter vMotion for large VMs>, these settings should be reverted to default once the activity has completed.

Configuring timeout settings for vpxd 🔝

To change the timeout value in the vCenter Server, and update /etc/vmware-vpx/vpxd.cfg in the vCenter Server Appliance (VCSA) using these steps:

Open an SSH connection to the VCSA and login with the root user.

Create a backup copy of the vpxd.cfg:

# cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg_backup

Edit the original file using the VI editor:
```
# vi /etc/vmware-vpx/vpxd.cfg
```
To increase the timeout values for general tasks - such as virtual machine migrations - add the following timeout parameter in the structure:
```
<config>
  ...
  <task>
    <timeout>10800</timeout>
  <task>
  ...
</config>
```
Note: The value is in seconds. The above example uses a 3 hour timeout, but the actual value can be changed, depending on the specific requirements.

To increase the SOAP layer blocking call timeout, add the following setting:

<config>
  ...
  <vmomi>
    <soapStubAdapter>
      <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
    </soapStubAdapter>
  </vmomi>
  ...
</config>

Save the changes and exit the editor with :wq!
Restart the vpxd service with:
```
# service-control --stop vmware-vpxd && service-control --start vmware-vpxd
```
For more information, see Stopping, starting or restarting VMware vCenter Server Appliance services.

Configuring timeout settings for vpxa in ESXi 7.0 Update 3 and later 🔝

Open an SSH connection with the ESXi host and log in using the root user.
Export the current settings for the vpxa service from the configstore using the following command:
```
# /bin/configstorecli config current get -c esx -g services -k vpxa -outfile current.json
```
Create a copy of the file:
```
# cp current.json change.json
```
Edit the file using the VI editor:
```
# vi change.json
```
Use the following entries to apply non-default timeout settings:
1. To change the general task timeout (e.g. to 10,800 seconds):
```
"task": {
      ...
      "timeout": 10800,
   },
```
2. For changing the VMACore SOAP timeout (e.g. to 10,800 seconds):
```
"vmacore": {
   ...
   "soap": {
      "session_timeout": 10800
   },
   ...
},
```
3. To alter the vmotion vm_id_acquire_timeout timeout for the time that vCenter Server waits to capture the virtual machine's ID at ESXi destination (e.g. to 600 seconds):
```
"vmotion":{
	  ...
   "vm_id_acquire_timeout": 600
   ...
 },
```
Save the file and exit the editor with :wq!
Run the following command to apply the changed configuration:
```
# /bin/configstorecli config current set -c esx -g services -k vpxa -infile change.json
```
A successful update can be recognized if the response is:
```
Set: completed successfully
```
Restart the vpxa service:
```
/etc/init.d/vpxa restart
```

Validate that the current configuration settings have applied the changes:

# /bin/configstorecli config current get -c esx -g services -k vpxa -outfile tmp_current.json

Configuring timeout settings for vpxa in ESXi 7.0 Update 2 and earlier versions 🔝

Open an SSH connection with the ESXi host and log in using the root user.

Create a backup copy of the vpxa.cfg file:

# cp /etc/vmware/vpxa/vpxa.cfg /etc/vmware/vpxa/vpxa.cfg_backup

Edit the file using the VI editor:
```
# vi /etc/vmware/vpxa/vpxa.cfg
```

Use the following entries to apply non-default timeout settings:

To change the general task timeout (e.g. to 10,800 seconds):

<config>
  ...
  <task>
    <timeout>10800</timeout>
  </task>
  ...
</config>

For changing the SOAP layer blocking call timeout (e.g. to 10,800 seconds):

<config>
  ...
  <vmomi>
    <soapStubAdapter>
      <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
    </soapStubAdapter>
  </vmomi>
  ...
</config>

To alter the vmotion vm_id_acquire_timeout timeout (e.g. to 600 seconds):

<config>
  ...
  <vpxa>
    ...
    <vmotion>
      <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
    ...
  </vpxa>
  ...
</config>

Save the file and exit the editor with :wq!
Restart the vpxa service:
```
/etc/init.d/vpxa restart
```