vCenter Server frequently disconnects and reconnects in VMware Cloud Director with the error message: "Error caused by vCenter Server disconnect."

Products

• VMware Cloud Director

Issue/Introduction

1. The vCenter Server repeatedly disconnects and reconnects every few seconds in VMware Cloud Director (vCD).
2. From VMware Cloud Director logs /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, the following entries are seen:

pc-activity-pool-532744   | JobManager | Add activity log: <JOB_VC_START_CONNECTION> <XXXX-vc01(com.vmware.vcloud.entity.vimserver:XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX)> |
pc-activity-pool-532744   | CJob| updateFailedJob(com.vmware.ssdc.util.LMException) with locale=null |
com.vmware.ssdc.util.LMException: Error caused vCenter Server disconnect
        at com.vmware.ssdc.backend.vimeventlistener.VirtualCenterListener.onVirtualCenterConnectionStateChanged(VirtualCenterListener.java:108)
        at com.vmware.vcloud.inventory.events.impl.VirtualCenterConnectionChangedEvent.dispatchTo(VirtualCenterConnectionChangedEvent.java:39)
        at com.vmware.vcloud.inventory.events.impl.EventManagerImpl.dispatchEvent(EventManagerImpl.java:514)

Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
Detail: Process 30906 waits for ShareLock on transaction 1571535272; blocked by process 330.
Process 330 waits for ShareLock on transaction 1571535282; blocked by process 30906.
  Hint: See server log for query details.
  Where: while updating tuple (15,1) in relation "property_map"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)

Environment

VMware Cloud Director 10.5.x and later

Cause

When updates are made to the property_map table from the channels LOW_PRIORITY and IAAS_OTHER simultaneously, a deadlock will occur. After three retries, the transaction fails, leading to VMware Cloud Director disconnecting the vCenter and then reconnecting.

Resolution

This issue has been resolved in VMware Cloud Director 10.6.1

Workaround:

To prevent these disconnects, disable the low priority channel. Disabling this channel does not have a critical impact.

Steps to disable low priority channel:

1. /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.pc.lowpriority.channel.enabled -v false
2. Reconnect vCenter in vCD

Note: vCD service restart is not needed.