Error "Operation canceled due to an unexpected error" when attempting Test Failover operations

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to perform a Test Failover within VMware Cloud Director Availability (VCDA), the operation frequently fails.
• Issue started when the Storage System went down
• /opt/vmware/h4/replicator/log/replicator.log shows following error messages: 

  WARN - [########-####-####-####-############] [offline-rtr-1] c.v.h.m.services.ReplicatorConnector     : Unable to reconnect to replicator: ########-####-####-####-############
  WARN - [########-####-####-####-############] [health-check-5] c.v.h4.common.tunnel.TunnelConnector     : No tunnels registered yet!

Environment

VMware Cloud Director Availability 4.7.X

Cause

The issue caused by the inability of the VCDA replicator service to establish or restore a secure communication tunnel with its peer component target hosts following the Storage System failure and left the VCDA services in a stale state.

Resolution

To resolve the issue apply the following steps:

1. SSH to the VCDA Manager and run the following command:

   systemctl restart manager

2. SSH to the  Replicator appliances and restart the replicator service with command:

   systemctl restart replicator