Following snapshot revert of vCenter, ESXi hosts are disconnected

Products

• VMware vCenter Server

Issue/Introduction

After reverting safe snapshots on vCenter server(s), one or more ESXi hosts are disconnected in inventory.

Environment

vCenter server and ESXi 7.0+

Cause

To understand root cause, one must understand how vCenter authenticates to hosts for communication/sync.  Authentication is achieved by way of a user account (vpxuser).  The password for vpxuser is rotated every 30 days by default, and the account is created at the time of initial connection to vCenter.  (for reference, see  How to modify the default expiry time for the vpxuser account)

 

Below, see this example:

 

Scenario:  There is a planned maintenance to patch a vCenter server

 

1.  In preparation, you have taken a snapshot of your vCenter, or all vCenters in the ELM group necessary (see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice)

2.  You begin the patching process of vCenter

3.  The vpxuser account password is automatically rotated in the background (as designed).  This updated in the vCenter database and on the host.

4.  The patching process fails for any reason, and you decide to roll back your vCenter(s).  As part of reverting the vCenter server to snapshot, vCenter database is rolled back as well.

 

vCenter is now using the old vpxuser password compared to the host.  This causes them to show as disconnected in the vCenter inventory.  When reconnecting them and providing the root password, the vpxuser is reset.

Resolution

Reconnect the ESXi hosts in inventory.  For instructions, see Disconnecting and Reconnecting an ESXi Host.