VMware Cloud Director intermittently disconnects from vCenter Server; tasks fail with the error: "The operation failed because vCenter Server 'vCenter Name' is not connected."

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director tasks intermittently fail with the following error message: The operation failed because vCenter Server "<vCenter Name>" is not connected.
  
  
  
• The /opt/vmware/vcloud-director/logs/vclistener.log file contains entries similar to:
  
  
  

  YYYY-mm-dd HH:MM:SS.zzz | WARN     | pc-activity-pool-#   | PropertyCollectorReader        | [Activity Execution] waitForUpdatesEx failed, will reconnect, vc-id=########-####-####-####-############, vc-runtime-id=########-####-####-####-############, job-id=########-####-####-####-############, message=null - Handle: urn:uuid:########-####-####-####-############ | activity=(com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader,urn:uuid:########-####-####-####-############)
  (vmodl.query.InvalidCollectorVersion) {
     faultCause = null,
     faultMessage = null
  }
   ::
  YYYY-mm-dd HH:MM:SS.zzz | DEBUG    | pc-activity-pool-#   | PropertyCollectorReader        | [Activity Execution] Disconnecting vc connection, vc-id=########-####-####-####-############, vc-runtimeid=########-####-####-####-############ - Handle: urn:uuid:########-####-####-####-############, Current Phase: PropertyCollectorReader$ReadPhase | activity=(com.vmware.vcloud.vimproxy.internal.impl.PropertyCollectorReader,urn:uuid:########-####-####-####-############)

Cause

VMware Cloud Director retrieves properties from vCenter Server through multiple channels. In rare instances, if these update processes conflict, the operation fails, and VCD attempts to re-establish the connection to vCenter Server.

Resolution

Starting from VMware Cloud Director 10.6.1, an option to adjust the retry interval was introduced to address this issue.
Adjust the following configuration setting to 3000 ms and verify if the symptoms persist:

cmt manage-config -n propertyCollector.deadlock.retry.sleep -v 3000

Note: Node restart is NOT required for this change to take effect.