ESXi hosts fail to enter maintenance mode in VCSA

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

 

• ESXi hosts fail to enter maintenance mode with the error "failed to enter namespaces maintenance mode on the node"
  
  
• WCP service failing to start.
  
  
• /var/log/vmware/eam/eam.log

YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | vim-monitor | VcConnection.java | 217 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

•  /var/log/vmware/wcp/wcpsvc.log

YYYY-MM-DDTHH:MM:SS.MSZ  error wcp [opID=EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later.

• VCLS VMs are not being created under vCenter clusters.

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

This can occur due to a discrepancy between the "vpxd-extension" certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension.

Resolution

Update the certificate for the Extension in VPXD by following any of below options: 

Option 1 - Update extensions using vCert script

vCert - Scripted vCenter expired certificate replacement

1. Execute following command to update vpxd-extension thumbprint
   
   

   ./vCert.py --run config/manage_cert/op_manage-vc-ext-thumbprints.yaml
============================================================================================
Please enter a Single Sign-On administrator account [[email protected]]: <-- Press Enter to use the default
Please provide the password for [email protected]: <-- Enter the password for <[email protected]>

   :::

   ------------------------!!! Attention !!!------------------------
Mismatched thumbprints detected.

   Update extension thumbprints? [n]: y   <-- Type 'y' and press Enter
============================================================================================

   
   
   
2. Restart eam service
   
   service-control --restart eam

 

 

Option 2 - Update extensions using fixcerts script

Update the extension's certificate using fixcerts script:

• Download the fixcerts script from :

Replace certificates on vCenter server using the Fixcerts script

• Copy the downloaded script to VCSA
• Execute the script using below arguments to update the extensions

python fixcerts.py update --ExtensionType all

Sample screenshot:

 

Note:

• eam, rbd & imagebuilder services will be restarted if it is in Running state. In above example, restart of rbd and Imagebuilder services are skipped as those services were in Stopped state before executing the script.
• More details about fixcerts script available in How to Replace Expired Certificates on vCenter Server using Fixcerts Python Script

Option 3 - Update extensions with the following article, EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates