Cloud Director Services showing as vcloud_portal_backend DOWN status

Products

• VMware Cloud Director

Issue/Introduction

Host CPU Performance may be degraded leading vCloud Director Cells to perform poorly and can go down.

An example of logs messages that will be seen in the /opt/vmware/vcloud-director/logs/cell.log

"Messaging Topology Cell Expirer" #242 daemon prio=5 os_prio=0 cpu=0.72ms elapsed=8.39s tid=0x00007f06b0033800 nid=0x5e2 waiting on condition  [0x00007f06711e5000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x0000000709815240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1177)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
        at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:829)

"JMX Monitor ThreadGroup<main> Executor Pool [Thread-3]" #243 daemon prio=5 os_prio=0 cpu=0.92ms elapsed=4.29s tid=0x00007f06f001f800 nid=0x5e3 waiting on condition  [0x00007f06f55dc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x0000000702e83e48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:2123)
        at java.util.concurrent.LinkedBlockingQueue.poll([email protected]/LinkedBlockingQueue.java:458)
        at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1053)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:829)

 

Environment

Cloud Director

Cause

The messages in the cell.log can indicate that there is a Host performance issue. The errors above was determined to be caused by CPU contention on the Host, indicated by a CPU Ready time average of 107,154 ms

Resolution

This indicates an issue on the ESXi Host and will need to be investigated further to determine and resolve the cause.