To fix the issue and resume the backups. And the snapshots to work again and resume the scheduled jobs.
All snapshots from the protected site have failed, and VCDR is unable to communicate with vCenter. Scheduled snapshots for all protection groups have failed, resulting in a critical status for the protection group, accompanied by the error: "The last snapshot of protection group xxx failed because it did not include any VMs."
In the event log, the following error appears for one of the protection groups:
Target: pg_Server
Description: Error taking snapshot for protection group 'pg_Server'. Internal error: RPC service unresponsive. Task ID: drc-xxxxxxw-xxxxx-xxxx-xxxx-0xxxx0xxxx.
When attempting to perform a manual snapshot, the same error persists. Additionally, creating a new protection group results in the browser for the protected vCenter remaining "loading," indicating that VCDR is unable to communicate with vCenter.
TROUBLESHOOTING:
The error "RPC service unresponsive" may indicate a communication issue with the DRC Connector VM. The connector VMs should be rebooted, and then allow 30 minutes before checking if you can create/edit the protection group and take a manual snapshot.
Also, verify the network connectivity of all DR connectors by following the instructions in the document:
DRC Network Test — drc network test --scope local
A change in the on-premises environment has resulted in a firewall block that is preventing communication with the VCDR service.
After removing the firewall block, the snapshots function correctly, and scheduled jobs can resume.
Impact/Risks:
There is a production impact; the customer's sites are unprotected, and backup/snapshot jobs cannot resume.