Embedded database performance is not stable after upgrading to VMware Cloud Director 10.6

Products

• VMware Cloud Director

Issue/Introduction

• After upgrading to VMware Cloud Director 10.6, the embedded postgres database performance is not stable.
• The performance from before the upgrade was consistent.
• Monitoring is done using Aria Operations suite. 
• After enabling trace logging as per the KB , the /opt/vmware/vcloud-director/logs/vclistener.log file shows a large number of entries of the following type: 
  TRACE    | pc-activity-pool-########    | ObjectUpdateProcessor          | vc-id=########-####-####-####-############, Handling update   ObjectUpdate of kind => 'modify', type => 'ClusterComputeResource' for object => 'domain-c########'
  
  
• The size of one of the VCD tables property_map is consistently increasing.

Environment

VMware Cloud Director 10.5

VMware Cloud Director 10.6

Cause

The spike in utilization was caused by cluster related updates being flooded from vCenter into the VCD. Due to this, one of the VCD tables (property_map) is being updated continuously, causing the number of dead records to keep increasing. The frequent updates to the same DB records would cause postgres compaction issues and leads to performance/timeout issues while reading data from postgres.

Resolution

Diagnose the issue as per the KB393588. Currently there is no resolution.

Known impacts of this issue are:
Case 1: Size of DB increase can cause cell to crash because of insufficient disk space
Case 2: Performance degradation. This is  mainly because of performance issue in querying the table

As a workaround, the cluster related updates causing the flooding can be moved to a lower priority channel to prevent postgres bloating. For assistance, contact technical support and note this Knowledge Article ID (398028) in the problem description. For more information, see How to Submit a Support Request.