Exit Standby Mode task times out and the host status is not updated in vCenter Server

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When powering on an ESX/ESXi host that is in the Standby mode, you experience these symptoms:

• The Exit Standby Mode task fails
• You see the error:
  
  Operation timed out
   
• The host remains in a Standby state in vCenter Server 4.x
• You can connect directly to the ESX/ESXi host with the vSphere Client.
• ESX/ESXi host responds to pings
• Attempting to power on the host through vCenter Server fails

Environment

VMware vSphere ESXi 5.1
VMware vCenter Server 5.0.x
VMware vCenter Server 5.1.x
VMware vCenter Server 4.0.x
VMware vCenter Server 4.1.x

Cause

This occurs when the Exit Standby Mode task in vCenter Server times out. In this case, vCenter Server becomes out of sync with the actual state of the host. The host is in Standby mode in vCenter Server, whereas it is actually powered on.

Resolution

To prevent this issue from occurring, increase the vCenter Server timeout period. For more information, see Extend the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager (1017253).

 

This allows the vCenter Server task to finish successfully and update the power state of the host.

To workaround this issue, update the vCenter Server database to reflect the correct host status.

 

To update the vCenter Server database to reflect the correct host status:

 

Note: Take a backup your vCenter Server database before implementing this workaround.

1. Stop the VMware VirtualCenter Server service.  
2. Run this command from vCenter Server SSH session to login to vPostgres database.
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
    
3. Run this SQL statement to find the ID of the host:
   
   select id from vpx_entity where name like '%name of host%'
    
4. Run this SQL statement to check the power state of the host:
   
   select power_state from vpx_host where id = <output 2 from step> </output>
   
   If the power state is 2, vCenter Server believes that the host is in Standby mode.
    
5. To put the host in a powered on state, run this update statement:
   
   update vpx_host set power_state = 0 where id = <output 2 from step> </output>
    
6. Start the VMware VirtualCenter Server service.
7. Ensure the ESX/ESXi host is powered on.
8. Disconnect and reconnect the ESX/ESXi host in vCenter Server if the host is not responding or disconnected.

Additional Information

Extending the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager