Any of the vCenter servers in Linked Mode report slow performance and an appliance CPU utilization hits 100%

Products

• VMware vCenter Server

Issue/Introduction

• Slowness while accessing vCenter server UI
• In performance charts of the vCenter server VM, CPU utilization of the vCenter reaches 100%
• In "top" command output on VC appliance , the vsphere-ui service consumes the highest CPU

# top  (press P (Shift+p), to order the results by CPU usage)

# top -H -p $(pgrep vsphere-ui)

• Threads showing high CPU utilization in top command, will log the thread dumps with name "I/O dispatcher" In /var/log/vmware/vsphere-ui/logs/vsphere-ui-runtime.log.stdout file:

e.g.

"I/O dispatcher 16" #474 prio=5 os_prio=0 tid=0x00007fb55427b800 nid=0x313a runnable [0x00007fb44e860000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007403d3820> (a sun.nio.ch.Util$3)
        - locked <0x00000007403d3810> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007403d36e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(Thread.java:750)

"I/O dispatcher 2" #460 prio=5 os_prio=0 tid=0x00007fb554262800 nid=0x3114 runnable [0x00007fb44ecce000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007404b74b8> (a sun.nio.ch.Util$3)
        - locked <0x00000007404b74a8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007404b7380> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(Thread.java:750)

"I/O dispatcher 22" #480 prio=5 os_prio=0 tid=0x00007fb554287800 nid=0x3144 runnable [0x00007fb44e67a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x000000074039ef70> (a sun.nio.ch.Util$3)
        - locked <0x000000074039ef60> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000074039ee38> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(Thread.java:750)

Environment

vCenter Server 7.0 U3

Site recovery manager 9.0.2

VR (vSphere Replication) 9.0.2

Cause

When vCenters are in linked mode, all UI plugin registrations are accessible across all vCenters.

Hence, a vCenter without SRM configured can still manage a vCenter with SRM configured. As a result, the UI components of the SRM plugin are available on all vCenters.

Therefore, vCenters without SRM configured may still exhibit the issue.

Resolution

Engineering is aware of this issue and it would be fixed in a future release.

Workaround:

A. Issue can be resolved temporarily by restarting the vSphere-ui service.

B. Until the fix is rolled out, please follow the below workaround:

(Tested on vCenter server 7.0.3 U3r, U3s and U3t builds, as all of these builds use same vSphere Client versions)

1. Download the attached h5-bridge-webapp.war file from this KB

2. Backup the original h5-bridge-webapp.war
# cp /usr/lib/vmware-vsphere-ui/server/webapps/h5-bridge-webapp.war /usr/lib/vmware-vsphere-ui/server/webapps/h5-bridge-webapp.war.bak

3. Replace the original h5-bridge-webapp.war with the patched one.

4. Update owner to root (if needed) and permissions of h5-bridge-webapp.war
# chmod 755 /usr/lib/vmware-vsphere-ui/server/webapps/h5-bridge-webapp.war

5. Restart vsphere-ui service
# service-control --restart vsphere-ui

6. In case a revert is needed, restore the original /var/core/h5-bridge-webapp.war and restart vSphere-ui service.