VMware Cloud Director Appliance Performance Degradation Due to Repeated "Removing Cluster" Log Entries

Products

• VMware Cloud Director

Issue/Introduction

• General day-to-day operations may take longer than expected.
• Artemis cluster is fully formed.
• CPU and RAM usage is within normal parameters.
• You may see these entries in the vcloud-conrtainer-debug.log:

2024-10-12 03:22:18,719 | DEBUG    | pc-activity-pool-12 | PCEventProcessingActivity | Processing events blob, request-token=vc-id=12c5ab47-3b45-483a-8194-cd71f2a5e920;moref=host-12547, event-types=[VimUpdateEvent(leave,HostSystem,[vcId=12c5ab47-3b45-483a-8194-cd71f2a5e920, moref=host-12547])], latency-ms=2 | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:9d18b5c2-7e33-4f47-a9a8-0d9e4b73f233)
2024-10-12 03:22:18,720 | WARN     | pc-activity-pool-12 | HostListener | Host ([vcId=12c5ab47-3b45-483a-8194-cd71f2a5e920, moref=host-12547]) deleted from VC | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:9d18b5c2-7e33-4f47-a9a8-0d9e4b73f233)
2024-10-12 03:22:18,720 | DEBUG    | pc-activity-pool-12 | HostListener | Ignoring host not in model db '[vcId=12c5ab47-3b45-483a-8194-cd71f2a5e920, moref=host-12547]' | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:9d18b5c2-7e33-4f47-a9a8-0d9e4b73f233)

 

2024-10-12 03:24:35,631 | DEBUG    | pool-54-thread-3 | ClusterComputeResourcePropertyAggregatorImpl | Removing cluster [vcId=1fa9f239-5b62-4e8e-9f36-8395f1a82b6d, moref=domain-c56789] from pendingClusters of size: 0 | 
2024-10-12 03:24:35,631 | ERROR    | pool-54-thread-3 | ClusterComputeResourcePropertyAggregatorImpl | Current cluster [vcId=1fa9f239-5b62-4e8e-9f36-8395f1a82b6d, moref=domain-c56789] not found in pendingClusters of size: 0 | 
2024-10-12 03:24:35,631 | DEBUG    | pool-54-thread-3 | ClusterComputeResourcePropertyAggregatorImpl | The pendingClusters now has following clusters in it: [] |

Environment

VMware Cloud Director 10.3

Cause

• There is an extremely high number of entries in the logs but there's no sign that the host is actually being added.
• What this means is that vCenter is constantly sending messages about the host leaving, even though the host is still listed in the system and is in maintenance mode.

Resolution

• If hosts exist in the cluster and are in maintenance mode, exit them from maintenance mode.
• If no hosts exist in the cluster then refresh and reconnect the vCenter connection in VCD so as to refresh the _inv tables.