vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services.

Products

• VMware vCenter Server

Issue/Introduction

• VCLS VMs are not provisioned.
• This may cause issues with remediation, resulting in: "Error to remediate cluster, error for service ESX Agent Manager"
• ESX Agent Manger will have multiple error traces. Where the error: "EAM is Still loading from database".
  
  /var/log/vmware/eam/eam.log

YYYY-MM-DDTHH:MM:SS | WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:###########] Re-login failed, due to:
com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.

YYYY-MM-DDTHH:MM:SS | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.

Environment

vCenter Server 7.x

Cause

There is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension.

Resolution

Verify and Resolve the mismatched EAM extension thumbprint using one of the following methods:

Note: Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

• Method 1: Manual thumbprint extension update (using vCenter CLI)
  
  
  • Log in to the vCenter Server Appliance using SSH.
  • Run these commands to retrieve the vpxd-extension solution user certificate and key:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key



• • Step 1: Identify the vCenter Server's PNID

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

• • Step 2: Update the Extension Certificate

Run this command to update the extension's certificate with vCenter Server (Use the PNID obtained above after -s in following cmd)

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u [email protected]

Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator's password.

• • Step 3: Restart the VMware ESX Manager service with these commands:

service-control --stop vmware-eam && service-control --start vmware-eam

• Method 2: Update thumbprints using vCert script.
  
           Note: This script is intended to be used at the direction of Broadcom Support.
  

  • Download and Prepare the vCert Script from vCert - expired certificate replacement script

  • Upload it to your vCenter Server appliance.

  • Unzip and make it executable:
    
    unzip -q vCert-<version>.zip
    cd vCert<version>
chmod +x vCert.py

  • Launch the Script
    
    ./vCert.py

  • Navigate to Certificate Management
    
    From the main menu, choose:
    
    3. Manage Certificates 
    
    Then In the sub-menu, choose
    
    6. vCenter Extension thumbprints

    The script will check for mismatches between stored thumbprints and actual certificates.