"vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs" alert in vSphere client UI
search cancel

"vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs" alert in vSphere client UI

book

Article ID: 400261

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • You see a warning message in vSphere Client as "vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS"
  • Unable to remove vCLS VMs that are powered off. (Retreat mode does not clean up the vCLS VMs)
  • The vSphere Cluster Service status is unhealthy.
  • In the /var/log/vmware/eam/eam.log

    YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension

    YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80

    YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-########-####-####-####-#######, subject domain: vsphere.local

    YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

    YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error

    (vim.fault.InvalidLogin) {

    faultCause = null,

    faultMessage = null

    }

    YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10

    or

    YYYY-MM-DDTHH:MM:SS.MSZ |  WARN | vim-async-1 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:################] Re-login failed, due to:
    com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
    .
    .
    Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password."

    YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
    com.vmware.eam.EamRemoteSystemException: Unexpected error communicating with the vCenter server.
    .
    .
    Caused by: com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The issue is caused by a vCenter Extension Thumbprint mismatch. This can be validated using the new improved certificate management tool  vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. This tool helps to replace certificates with VMCA signed as well as custom CA signed certificates. 

To confirm the mismatch, follow these steps:

  1. Run the vCert script on your vCenter server.

  2. Acknowledge the snapshot and risks warning.

  3. Enter the administrator credentials.

  4. Select option 1 (view certificate info) from the menu.

The script will perform a check, and if a mismatch exists, the output will indicate this discrepancy.

Here’s an example of what you might see in the script output if a mismatch is present:

Resolution

Before proceeding with the steps below, take both a backup and a snapshot of the vCenter Server Appliance. If the vCenter is part of a Enhanced Linked Mode (ELM) replication setup, also take a backup or offline (powered off) snapshot of all replicating vCenter ELM nodes.

To resolve this issue,

Option 1: Update the vCenter Extension Thumbprints using the new improved certificate management tool  vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. This tool helps to replace certificates with VMCA signed as well as custom CA signed certificates. 

  • Run the vCert script on your vCenter server.

  • Acknowledge the snapshot and risks warning.

  • Enter the administrator credentials.

  • Select Option 3 to manage certificates.

  • Then, select Option 6 (vCenter Extension thumbprints) to check and update the vCenter Extension Thumbprint mismatch.

Option 2: Manually update the vpxd-extension certificate of vCenter Server by following below steps. 

  • Log in to the vCenter Server Appliance using SSH. 
  • Run this command to retrieve the vpxd-extension solution user certificate and key:
    • mkdir /certificate
    • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
    • /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
  • Edit the "<vCenter_Server_Hostname>" in the below command and run this command to update the extension's certificate with vCenter Server.
    • python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter_Server_Hostname> -u [email protected]

NOTE: The default user and domain is [email protected]. Change the domain to match the environment's vCenter SSO. When prompted, type in the [email protected] password.

  • Restart the VMware ESX Agent Manager service with these commands: 
    • service-control --stop vmware-eam
    • service-control --start vmware-eam
Powered by Wolken Software