[VMC on AWS] One or more Virtual machine in the protection group may fail recovery test

Products

• VMware Live Recovery
• VMware Cloud on AWS

Issue/Introduction

To help identify the cause and resolution steps to resolve the issue.

Symptoms:

• Some virtual machines in the protection group may fail test recovery. 

opt/vmware/hms/logs/hms.log 

 INFO  hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-6] (..response.filter.I18nActivationResponseFilter) [operationID=########-####-####-####-########7378 HMS-2809966,sessionID=3398820A48DADF7241E91EC6E5392533850168C9CE1BCF3F36B049402084E23DB8F8E8976296821452A5002B46DA5D265A71E76AC83ECA4D8069BCB1FAB980B6] | The localized message is: A generic error occurred in the vSphere Replication Management Server. Exception details: 'Failed storing configuration state and error'.

 INFO  hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-6] (..response.filter.I18nActivationResponseFilter) [operationID=ae649dfd-957a-4a97-91e8-a157c
2dd7378-HMS-2809966,sessionID=3398820A48DADF7241E91EC6E5392533850168C9CE1BCF3F36B049402084E23DB8F8E8976296821452A5002B46DA5D265A71E76AC83ECA4D8069BCB1FAB980B6] | The localized message is: Object 'GID-########-####-####-####-########8d7a' is locked by another ongoing operation in vSphere Replication Management Server. Try again later.

• Unprotecting and re-protecting the affected virtual machines does not help
• Increasing the time-out values in SRM does not help KB94060
• Running a manual sync for the affected Virtual Machines may fail with error "Virtual machine is not configured for replication"
• Unregistering and re-registering the affected virtual machines does not resolve the problem

Cause

The recovery tests may fail due to timeouts and might be cancelled by the user. However, the failed test task might still be running in the background on SRM and will be locking the Virtual Machine which results in subsequent tasks initiated to fail.

Resolution

• Engage VMware Support to have the hms and hbr services to be restarted in the recovery side 
• Remove the replication on the affected Virtual Machines and configure it again. Wait for the replication to finish. 
• Confirm that the affected Virtual Machines are in healthy state in the protection group. 
• Attempt a test recovery.

Additional Information

Impact/Risks:
Test recovery may fail for some Virtual machines and cannot be protected.