vCenter operation times out with the error: Operation failed since another task is in progress

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

The default time-outs in VMware Infrastructure (VI) Client may not be long enough for certain long operations, such as deleting snapshots. This article provides information on how to prevent these timeouts.

Symptoms:

• vCenter Server tasks time out or stop responding.
• If you are using VMware Consolidated Backup (VCB), the backup fails, but snapshots are left behind.
• Long warm or cold migrations time out.
• If the task runs for more than 15 minutes, vCenter Server forces it to time out, but the task continues the execution on ESXi/ESX host.
• Next attempt to re-run the task returns the error:
  
  Operation failed since another task is in progress
  
  
• Unable to delete snapshot.
• vCenter Server times out when deleting snapshots.
• Snapshots growing indefinitely.
• If an alarm is configured to send an email, you see the message:
  
  Alarm Host connection state on esx.example.com sent e-mail to [email protected]
  
  
• Operations that rely on vCenter Server connectivity, such as vMotion, Storage vMotion, VMware Site Recovery Manager, and VCB, fail.
• Lengthy VMware Update Manager tasks have also been known to time out.

Environment

VMware vSphere ESXi 
VMware vCenter Server 

Resolution

vCenter Server has a default 15 minute timeout for any task. Starting with the vCenter 2.5 Update 4 release, to prevent vSphere Client from displaying unnecessary timeout error messages, you can configure the timeout values by editing the vpxd.cfg file and the vpxa.cfg file of the source and destination ESXi host.

 

Note: If you are using VCB and your backup failed due to a timeout, check your virtual machine for a backup snapshot that has been left behind.

Lengthy Tasks which Time Out

When a task is reported to timeout within vCenter, the task may continue to run in the ESXi host level. Certain tasks (such as a snapshot consolidation) may take a long time to complete and should not be interrupted. For information on how to identify tasks within a VMware ESXi host, see Collecting information about tasks in VMware ESXi (1013003).

 

Note: In the case of snapshot consolidation, even though the vSphere Client timeout occurs, the operation on the ESXi/ESX host is still running. You can verify by observing the .vmdk file for the virtual machine. It is updated every minute which means the delta files are being committed to the .vmdk file.

 

vCenter Server Timeout Settings

To change the timeout value in the vCenter Server, update vpxd.cfg on vCenter and vpxa.cfg on the ESXi/ESX:

 

1. Log in to the vCenter Server with the appropriate permissions.
    
2. Open the vpxd.cfg file in a text editor. The default location for the file is:
   
   C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
   
   
    
3. For Windows 7 and Windows 2008, the default location for the file is:
   
   C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
   
    
4. To increase the timeout values for the virtual machine migration task, add the following timeout parameter in the vpxd.cfg file:
   
   <config>
   ...
   <task>
   <timeout>10800</timeout>
   </task>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
   
   
    
5. To increase the SOAP layer blocking call timeout, add the following values in the vpxd.cfg file:
   
   <config>
   ...
   <vmomi>
   <soapStubAdapter>
   <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
   </soapStubAdapter>
   </vmomi>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds, or 3 hours.
   
   
6. Restart the vCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).

ESXi/ESX timeout settings

1. Log in to the ESXi host as root via the console or an SSH session. For more information about ESXi, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807). 
    
2. Open the vpxa.cfg file in a text editor.
   
   By default, this file is located at:
   
   

   • ESX - /etc/opt/vmware/vpxa/vpxa.cfg

   • ESXi - /etc/vmware/vpxa/vpxa.cfg

   Note: ESXi and ESX hosts come with open source tools which can be used for troubleshooting. The nano editor can be used in ESX, while the vi editor can be used in ESXi or ESX. For more information on these tools, see GNU nano and Vim.
   
    

3. To increase the timeout values for the virtual machine migration task (both source and destination hosts), add the following timeout parameter in the vpxa.cfg file:
   
   <config>
   ...
   <task>
   <timeout>10800</timeout>
   </task>
   ...
   </config>
    

4. To increase the SOAP layer blocking call timeout, add these values in the vpxa.cfg file :
   
   <config>
   ...
   <vmomi>
   <soapStubAdapter>
   <blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
   </soapStubAdapter>
   </vmomi>
   ...
   </config>
   
   Note: The value 10800 can be changed based on your requirements. This example uses 10800 seconds or 3 hours.

5. Configure the timeout value for the time that vCenter Server waits to capture the virtual machine's ID at ESXi destination. Add a new configurable parameter in the vpxa.cfg file:
   
   <config>
   ...
   <vpxa>
   ...
   <vmotion>
   <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
   </vmotion>
   ...
   </vpxa>
   ...
   </config>
   
   
   Note: The value 600 can be changed based on your requirements. This example uses 600 seconds or 10 minutes.

6. Restart the vmware-vpxa service on the ESXi/ESX host. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).

Additional Information

 

Restarting the Management agents in ESXi
Unable to connect to an ESX host using Secure Shell (SSH)
How to stop, start, or restart vCenter Server services

Collecting information about tasks in VMware ESXi