Replication failing with RPC Null Error in VMware Live Cyber Recovery Protection Groups

Products

• VMware Cloud on AWS

Issue/Introduction

In VLCR GUI, replication tasks appear to be failing with error  "RPC: (null)" error as seen below

Environment

VMware Live Cyber Recovery 7.27.x

VMware Live Cyber Recovery 7.26.x

Cause

This is caused by too many concurrent API calls made to VLCR Connector VM causing service(s) to get into non-responding state.

Resolution

To resolve this issue follow below steps: 

1. SSH to connector VM using admin account. 
2. Run 'entersupportmode' followed by 'dev root enable' to get root password.
3. SSH to connector again as root using provided credentials. 
4. Run 'docker ps' to list the current running processes on connector. 
5. Find the container ID for the  'drc_SysMgmt' service. 
6. Run 'docker restart <container ID>' to restart the sysmgmt process. 
7. Run 'docker ps' to verify that the service came back online. 

8. Take another manual snapshot for the PG that failed with the RPC (null) error, to confirm the snapshot task is successfull. 

Note: If there are multiple connector VMs, restart the service on all the connector VMs.

 

Alternatively, reboot all the connector VMs, this should restart all the docker containers. Attempt to take a manual snapshot after the connectors are back online.