vCenter Server frequently disconnects and reconnects in VMware Cloud Director with the error message: "Error caused by vCenter Server disconnect."

Products

• VMware Cloud Director

Issue/Introduction

1. The vCenter Server repeatedly disconnects and reconnects every few seconds in VMware Cloud Director (vCD).
2. From VMware Cloud Director logs /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, the following entries are seen:

pc-activity-pool-532744   | JobManager | Add activity log: <JOB_VC_START_CONNECTION> <XXXX-vc01(com.vmware.vcloud.entity.vimserver:XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX)> |
pc-activity-pool-532744   | CJob| updateFailedJob(com.vmware.ssdc.util.LMException) with locale=null |
com.vmware.ssdc.util.LMException: Error caused vCenter Server disconnect
        at com.vmware.ssdc.backend.vimeventlistener.VirtualCenterListener.onVirtualCenterConnectionStateChanged(VirtualCenterListener.java:108)
        at com.vmware.vcloud.inventory.events.impl.VirtualCenterConnectionChangedEvent.dispatchTo(VirtualCenterConnectionChangedEvent.java:39)
        at com.vmware.vcloud.inventory.events.impl.EventManagerImpl.dispatchEvent(EventManagerImpl.java:514)

Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
Detail: Process 30906 waits for ShareLock on transaction 1571535272; blocked by process 330.
Process 330 waits for ShareLock on transaction 1571535282; blocked by process 30906.
  Hint: See server log for query details.
  Where: while updating tuple (15,1) in relation "property_map"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)

Environment

VMware Cloud Director 10.5.x and later

Cause

When updates are made to the property_map table from the channels LOW_PRIORITY and IAAS_OTHER simultaneously, a deadlock will occur. After three retries, the transaction fails, leading to VMware Cloud Director disconnecting the vCenter and then reconnecting.

Resolution

This is a known issue, which has been addressed in future release of Cloud Director.

Workaround:

To prevent these disconnects, disable the low priority channel. Disabling this channel does not have a critical impact.

Steps to disable low priority channel:

1. /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.pc.lowpriority.channel.enabled -v false
2. Reconnect vCenter in vCD

Note: vCD service restart is not needed.