Replication failing with RPC Null Error in VMware Live Cyber Recovery Protection Groups

Products

• VMware Cloud on AWS

Issue/Introduction

In VLCR GUI, replication tasks appear to be failing with error  "RPC: (null)" error as seen below

Environment

VMware Live Cyber Recovery 7.27.x

VMware Live Cyber Recovery 7.26.x

Cause

This is caused by too many concurrent API calls made to VLCR Connector VM causing service(s) to get into non-responding state. It may also occur due to an older version of connector appliance in use post upgrade to the VLCR backend components. 

Resolution

To resolve this issue follow below steps: 

1. SSH to connector VM using admin account. 
2. Run 'entersupportmode' followed by 'dev root enable' to get root password.
3. SSH to connector again as root using provided credentials. 
4. Run 'docker ps' to list the current running processes on connector. 
5. Find the container ID for the  'drc_SysMgmt' service. 
6. Run 'docker restart <container ID>' to restart the sysmgmt process. 
7. Run 'docker ps' to verify that the service came back online. 

8. Take another manual snapshot for the PG that failed with the RPC (null) error, to confirm the snapshot task is successfull. 

Note: If there are multiple connector VMs, restart the service on all the connector VMs.

Alternatively, reboot all the connector VMs, this should restart all the docker containers. Attempt to take a manual snapshot after the connectors are back online. 

If the issue is seen post upgrade to the VLCR orchestrator and cloud filesystem, then verify that the connector appliance version matches the VLCR version. You can check the connector appliance version by clicking on connector info. Redeploy the connector if its not matching the VLCR Orchestrator version.