Error: "Partially Connected" observed when viewing the Multisite Association Status in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• "Partially Connected" status shows for Multisite association in Cloud Director.
• Multisite not working on Cloud Director. One site shows as "Partially Connected".
• The association is lost on one of the cloud director instance and you cannot see any organization in the cloud director site.
• User does not get an output  for job 'MultisiteAssociationStatusSynchronizer' when running  /opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -tt
  
  
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log or vcloud-container-info.log , the provider might see the following errors : 

com.vmware.vcloud.multisite.query.QueryExecutor$AbstractExecutor.extractResultsFromResponseFuture(QueryExecutor.java:476)
com.vmware.vcloud.multisite.query.QueryExecutor$AbstractExecutor.execute(QueryExecutor.java:252)
com.vmware.vcloud.multisite.query.OpenApiQueryExecutor.executeQuery(OpenApiQueryExecutor.java:53)
com.vmware.vcloud.multisite.query.QueryExecutor.execute(QueryExecutor.java:111)
com.vmware.vcloud.api.rest.providers.filters.openapi.MultisiteOpenApiFilter.handleMultisiteRequest(MultisiteOpenApiFilter.java:77)
com.vmware.vcloud.api.rest.providers.filters.MultisiteFilter.filter(MultisiteFilter.java:145)

Environment

VMware Cloud Director 10.6.0.1
VMware Cloud Director 10.5.1.1

Cause

This issue occurs when the MultisiteAssociationStatusSynchronizer is not running. This is responsible for setting the state of the multisite link.
MultisiteAssociationStatusSynchronizer should be visible as an active scheduled job on at least one cell of the Cloud Director.

For example:

Currently active scheduled jobs:+--------------------------------------+-------------------------+-------------------------+-----------------+----------------------------------------------+
| UUID                | Previous Start Time  | Next Start Time    | Status     | Job Name                  |        +--------------------------------------+-------------------------+-------------------------+-----------------+----------------------------------------------+
  | <SITE_UUID> | 2022-01-11 13:53:43.936 | 2022-01-11 13:54:49.829 | STATUS_QUEUED | MultisiteAssociationStatusSynchronizer   

Resolution

This issue involves Database modifications, submit a Support Request with Broadcom Support for further assistance.

Note: You will need to ensure that a change window is organized, as the services of Cloud Director will need to be restarted, and some information may need to cleared from the Cloud Director Database.