Error: vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services
search cancel

Error: vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services

book

Article ID: 407460

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Distributed Resource Scheduler (DRS) does not work on the specific cluster in vCenter Server. vSphere Cluster Services (vCLS) virtual machines are not created after you toggle DRS or vCLS retreat mode.
  • The cluster summary page displays the following alarm: "vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS".

  • Disabling and re-enabling vSphere High Availability (HA) does not resolve the issue. Restarting all Management Agents within the cluster does not resolve the unhealthy status.
  • In the vCenter Server /var/log/vmware/eam/eam.log the following entries occur,
[YYYY-MM-DDTHH:MM:SS] | ERROR | vim-monitor | VcListener.java | 124 | An unexpected error in the changes polling loop
com.vmware.eam.EamRemoteSystemException: Client error communicating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:103) ~[eam-server.jar:?]
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140) ~[eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118) [eam-server.jar:?]
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_452]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_452]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_452]
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:61) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:267) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:56) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120) ~[vlsi-core.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:52) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:128) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322) ~[vlsi-client.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195) ~[vlsi-client.jar:?]
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source) ~[?:?]
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101) ~[eam-server.jar:?]
      
[YYYY-MM-DDTHH:MM:SS]|  INFO | vim-monitor | VcListener.java | 125 | Full stack trace: com.vmware.eam.EamRemoteSystemException: Client error communicating with the vCenter server.
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:103)
        at com.vmware.eam.vim.server.impl.VimRoot.currentTime(VimRoot.java:78)
        at com.vmware.eam.vc.VcListener.main(VcListener.java:140)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:118)
        at com.vmware.eam.vc.VcListener.call(VcListener.java:58)
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58)
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:61)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:267)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:56)
        at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:52)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:128)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
        at com.sun.proxy.$Proxy51.currentTime(Unknown Source)
        at com.vmware.eam.vim.server.impl.VimRoot.rootOperation(VimRoot.java:101)

 

  • In the vCenter Server /var/log/vmware/wcp/wcpsvc.log the following entries occur,

 [YYYY-MM-DDTHH:MM:SS] error wcp [eamlib/lister.go:84] [opID=EAMAgent] Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.
 [YYYY-MM-DDTHH:MM:SS] error wcp [informer/informer.go:129] [opID=EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later.

 

Environment

  • VMware vCenter Server 8.x
  • VMware ESXi 8.x

Cause

This issue occurs because the Domain Name System (DNS) configuration is inconsistent across the cluster. Several ESXi hosts are missing the required DNS domain suffix.

Resolution

To restore DRS functionality, configure the DNS domain suffix on all ESXi hosts in the cluster. The configuration can be applied through Host Profiles or manually through the vSphere Client.

Option 1: Update using Manual Configuration

  1. Navigate to Hosts and Clusters and select the ESXi host.
  2. Select the Configure tab.
  3. Go to Networking > TCP/IP Configuration.
  4. Select the Default TCP/IP stack and click Edit.
  5. Input the required domain suffixes in the "Domain field" and "Search Domains field". (use commas as delimiters for multiple entries).
  6. Click OK to save the changes.
  7. To restart services log in to the ESXi Shell as root.

     a. Run /etc/init.d/hostd restart and /etc/init.d/vpxa restart commands to restart the associated services.

Note: Repeat these steps for every host in the cluster to ensure consistency.

Warning: Do not use the services.sh restart command or restart management agents via the Direct Console User Interface (DCUI). Avoid this if vSAN, Link Aggregation Control Protocol (LACP), NSX, or shared graphics (vGPU, vDGA, vSGA) are in use.


Option 2: Update using Host Profile

  1. Navigate to Policies and Profiles > Host Profiles and select the desired profile.
  2. Click Edit Host Profile.
  3. Navigate to Networking configuration > Netstack instance > defaultTcpipStack > DNS configuration.
  4. Locate the Domain name and Search domains fields.
  5. Input the required domain suffixes (use commas as delimiters for multiple entries).
  6. Save the profile and Remediate the hosts in the cluster.
  7. To restart services log in to the ESXi Shell as root.

    a. Run  /etc/init.d/hostd restart and /etc/init.d/vpxa restart commands to restart the associated services.

Warning: Do not use the services.sh restart command or restart management agents via the Direct Console User Interface (DCUI). Avoid this if vSAN, Link Aggregation Control Protocol (LACP), NSX, or shared graphics (vGPU, vDGA, vSGA) are in use.

Powered by Wolken Software