ESXi host gets stuck while entering maintenance mode because the embedded vCLS VMs are not evacuated.

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host fails to enter maintenance mode, and the corresponding task appears stuck or hangs.
  
  

• This issue is caused by the embedded vCLS VMs (introduced in ESXi 8.0U3 or later) failing to automatically evacuate from the host.
  
  Note: This article applies to embedded vCLS VMs. For similar symptoms related to external vCLS VMs (pre-ESXi 8.0U3 versions), please refer to: ESXi hosts fail to enter maintenance mode in VCSA
  
  

• In /var/log/vmware/vpxd/vpxd.log on vCenter:
  
  [timestamp] warning vpxd[18668] [Originator@6876 sub=MoCluster opID=PodCrxMgr-domain-cXX-1882] Failed to disable vCLS Pod Crx; [vim.HostSystem:host-XX,<ESXi host name>],
 [vim.ClusterComputeResource:domain-c35,<Cluster Name>], N4Vpxd7Clients24UnauthenticatedExceptionE(Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> No messages!
--> )
  
  
  
• In /var/run/log/settingsd.log on the ESXi host:
  
  [timestamp] Er(11) settingsd[132451]: Auth_AuthenticateUserPAM:424: PAM failure - Authentication failure (7)
[timestamp] Er(11) settingsd[132451]: error [ConfigStore:a5a1a7a700] UserPass authentication failed for user: vpxuser
[timestamp] Er(11) settingsd[132451]: error [ConfigStore:a5a1a7a700] Method failed


Environment

ESXi 8.0U3 or later

Cause

This issue is caused by a password mismatch for the vpxuser account between vCenter Server and the ESXi host.

Resolution

To resolve this issue, resolve the vpxuser password inconsistency between vCenter Server and the ESXi host.

The underlying cause for this password mismatch can vary.

One potential cause and its corresponding solution are outlined in the following article:

vpxuser password change fails repeatedly