vCloud Director vApp Virtual Machine Monitoring Chart contains no data

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:
vcloud container-info.log contains errors similar to:

2021-10-02 10:05:12,019 | ERROR    | pool-jetty-989492         | JDBCExceptionReporter          | [pool-jetty-989492] Timeout: Pool empty. Unable to fetch a connection in 20 seconds, none available[size:75; busy:75; idle:0; lastwait:20000]. | requestId=<TASK_UUID>,request=GET https://vcloud.example/api/admin/network/<NETWORK_UUID>,requestTime=1633212312232,remoteAddress=<IP>:37314,userAgent=Apache-CXF/3.3.0,accept=application/*+xml;version 34.0

There also may be errors similar to:

2021-10-02 10:02:35,626 | ERROR    | auto-pool-CoordinatorJob-49-thread-1 | FailoverServiceManagerImpl     | Error during the completion of recovery for for failed cell <CELL_UUID> instance 1,993 | 
java.lang.NullPointerException

2021-10-05 19:31:15,287 | ERROR    | activemq-monitor-timer    | ActiveMQMonitorTimerTask       | Exception while sending echo message [echoString=3/5/21 7:31 PM, localTime=3/5/21 7:31 PM] to cell [CellSpec: cellId = <CELL_ID>, cellUuid = <CELL_UUID>x, cellInstanceId = 4433, cellName = vcloud01.example, isActive = true, productVersion = 10.1.2.16779297, productBuildDate = 2020-08-21 01:43:19.0, heartbeat = 2021-10-06 00:30:54 UTC, maxVcProxies = -1]
com.vmware.ssdc.backendbase.failover.FailoverMessageServiceException: javax.jms.JMSException: AMQ219010: Connection is destroyed
...
Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
...
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]
        ... 21 more
VMware Cloud Director 10.1.2.16779297, Time,vcloud.example.com (<IP>)
VMware Cloud Director 10.1.2.16779297, Time, vcloud.example.com (<IP>)

Environment

VMware Cloud Director 10.x

Cause

The stats feeder process is not running because the DB connections pool is exhausted.

Resolution

1. Increase the db connection pool as described in KB 59537

After services are up and running for 10 to 20 minutes, check to see if the Machine Monitoring chart contains data.

2. If problem persists clear the records from the failed_cells and scheduled_activity_jobs table as follows:

a. Backup the database prior to proceeding.
b. Run these commands on the database:
delete * from failed_cells
delete * from scheduled_activity_jobs
c. Restart the vmware-vcd service on the cells.
service vmware-vcd restart