PostgreSQL routinely causes CPU spikes requiring a reboot to resolve in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

Over the time CPU usage of a VMware Cloud Director primary database cell will reach >90%. This will be seen with errors in the logs such as:

YYYY-MM-DD HH:MM:SS | ERROR    | stats-activity-pool-#### | CustomCassandraStatsReceiver   | Error processing entity stats for vc:<vCenter UUID> | activity=(com.vmware.vcloud.metrics.sta
tsfeeder.StatsFeederCollectorActivity,urn:uuid:<Activity UUID>)
java.lang.NullPointerException
        at com.vmware.vcloud.statsfeeder.cassandra.CustomCassandraStatsReceiver.receiveStats(CustomCassandraStatsReceiver.java:##)
        at com.vmware.vcloud.metrics.statsfeeder.StatsFeederCollectorActivity$StatsFeederCollectorPhase.invoke(StatsFeederCollectorActivity.java:##)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:###)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:###)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:###)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:###)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:####)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:###)
        at java.base/java.lang.Thread.run(Thread.java:###)

 

Environment

VMware Cloud Director 10.x

Cause

The Cassandra DB integration with VMware Cloud Director is causing PostgreSQL queries to hang. This is causing PostgreSQL itself to consume more resources on the VM. 

Resolution

Remove the integration with Cassandra DB. On the cells run:

/opt/vmware/vcloud-director/bin/cell-management-tool cassandra --clean

Additional Information

Japanese KB: VMware Cloud DirectorにおけるPostgreSQLの高頻度なCPUスパイクを引き起こし、再起動が必要となる