vCenter operation times out with the error: Operation failed since another task is in progress

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

The default timeouts in vCenter may not be long enough for certain long operations, such as deleting snapshots. This article provides information on how to prevent these timeouts.

Symptoms:

• vCenter Server tasks time out or stop responding.
• If the backup fails, but still the task is running in the backend.
• Long warm or cold migrations time out.
• If the task runs for more than 15 minutes, vCenter Server forces it to time out, but the task continues the execution on ESXi host.
• Next attempt to re-run the task returns the error:
  
  Operation failed since another task is in progress
  
• Unable to delete snapshot.
• vCenter Server times out when deleting snapshots.
• Snapshots growing indefinitely.
• If an alarm is configured to send an email, you see the message:
  
  Alarm Host connection state on esx.example.com sent e-mail to [email protected]
  
  
• Operations that rely on vCenter Server connectivity, such as vMotion, Storage vMotion, Replication via SRM fail.
• Lengthy VMware Update Manager tasks have also been known to time out.

Environment

VMware vSphere ESXi 
VMware vCenter Server 

Resolution

vCenter Server has a default 15 minute timeout for any task. Starting with the vCenter 2.5 Update 4 release, to prevent vSphere Client from displaying unnecessary timeout error messages, you can configure the timeout values by editing the vpxd.cfg file and the vpxa.cfg file of the source and destination ESXi host.

 

Note: If you are using VCB and your backup failed due to a timeout, check your virtual machine for a backup snapshot that has been left behind.

Lengthy Tasks which Time Out

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted. For information on how to identify tasks within a VMware ESXi host, see Collecting information about tasks in VMware ESXi.

 

Note: In the case of snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. You can verify by observing the .vmdk file for the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.

 

vCenter Server Timeout Settings

To change the timeout value in the vCenter Server, update vpxd.cfg on vCenter.

 

1. Log in to the vCenter Server via SSH as root user.
   
   Note:
   a) Take backup of the vpxd.cfg file using below command
   cp vpxd.cfg vpxd.cfg.backup
   b) Or Take Snapshot of the vCenter Server (Offline Snapshots of the respective vCenter Servers if they are in Linked Mode).
   
   
    
2. Open the vpxd.cfg file in a text editor using below command.
   
   vi /etc/vmware-vpx/vpxd.cfg
   
   
    
3. To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file:
   
   <config>
   ...
   <task>
   <timeout>10800</timeout>
   </task>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
   
   
    
4. To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file:
   
   <config>
   ...
   <vmomi>
   <soapStubAdapter>
   <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
   </soapStubAdapter>
   </vmomi>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
   
   
5. Press Esc + : + wq! to save and exit vpxd.cfg file.
   
   
6. Restart the vCenter Server service using below command.
   service-control --stop && service-control --start
   For more information, see Stopping, starting, or restarting vCenter services.

ESXi timeout settings

1. Log in to the ESXi host as root via the console or an SSH session.
   
   
2. Open the vpxa.cfg file in a text editor using below command.
   vi /etc/vmware/vpxa/vpxa.cfg
   
   
3. To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file:
   
   <config>
   ...
   <task>
   <timeout>10800</timeout>
   </task>
   ...
   </config>
    

4. To increase the SOAP layer blocking call timeout, add these values in the vpxa.cfg file:
   
   <config>
   ...
   <vmomi>
   <soapStubAdapter>
   <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
   </soapStubAdapter>
   </vmomi>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds or 3 hours.

5. Configure the timeout value for the time that vCenter Server waits to capture the virtual machine's ID at ESXi destination. Add a new configurable parameter in the vpxa.cfg file:
   
   <config>
   ...
   <vpxa>
   ...
   <vmotion>
   <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
   </vmotion>
   ...
   </vpxa>
   ...
   </config>
   
   Note: The value 600 can be changed based on your requirements. This example uses 600 seconds or 10 minutes.

     6. Press Esc + : + wq! to save and exit vpxa.cfg file.

     7. Restart the vmware-vpxa service on the ESXi/ESX host. For more information, see Restarting the Management agents on an ESXi or ESX host.

Additional Information

Restarting the Management agents in ESXi
How to stop, start, or restart vCenter Server services
Collecting information about tasks in VMware ESXi