Error: "Socket read timeout." when performing the reverse of a vApp replication

Products

• VMware Cloud Director

Issue/Introduction

• When you reverse a vApp replication that was previously failed over using VMware Cloud Director Availability (VCDA), some of the VM replications reverse without issue but others fail with the error:
  
  Socket read timeout.
  
  
• In the /opt/vmware/h4/replicator/log/replicator.log file on the destination Replicator appliance, you see a message similar to:
  
  ERROR - [UI-########-####-####-####-############-r4735-hG-Arg-Ea-9h-Xk] [job-42] com.vmware.h4.jobengine.JobExecution     : Task ########-####-####-####-########8cb9 (WorkflowInfo{type='setupDestination', resourceType='replication', resourceId='H4-########-####-####-####-########51ba', isPrivate=false, resourceName='null'}) has failed

com.vmware.exception.ReadTimeoutException: https://vcenter.example.com:443/pbm/sdk invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.exception.converter.ClientExceptionConverter.convertException(ClientExceptionConverter.java:42)
        at com.vmware.vlsi.util.ExceptionConverterInterceptor.handleException(ExceptionConverterInterceptor.java:32)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:263)
        ...
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: https://example.com:443/pbm/sdk invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
        ... 29 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
        ...
  
  
• In the /var/log/vmware/vmware-sps/sps.log file on the destination vCenter Server, you see messages similar to:
  
  [pool-4-thread-15] INFO  opId=UI-########-####-####-####-############-r4735-hG-Arg-Ea-9h-Xk com.vmware.pbm.placement.impl.PlacementSolverImpl - Timer stopped: queryMatchingHub, Time taken: 55357 ms.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs because the PBM service takes too long to complete a placement query and Cloud Director Availability times out while waiting for a response from the vCenter Server.

Resolution

To resolve this issue, the performance issue on the destination vCenter Server needs to be resolved.

For example, if you are using Aria Operations and vSAN in the destination site you may want to upgrade to the latest release of Aria Operations to reduce the load it places on vCenter Server.

Workaround:

A restart of the vCenter Server can temporarily mitigate this issue. For more information, see Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services.