Protecting VM via HCX -DR fails with "Error Creating Group"

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

• While Protecting VM via HCX-DR [Disaster Recovery] , the following error is observed in /common/logs/admin/app.log :
  

  <timestamps> UTC [VsphereReplicationService_SvcThread-14697, T:'Replication Group Creator On VRS', R:protect-8214199d-###-####, J:0d007955-###-####, S:CREATE_GROUP, , TxId: da5f3cc6-###-####] INFO  c.v.v.h.s.v.j.VRSGroupMappingCreator- Encountered
  error while execution, starting error handling
  <timestamps> UTC [VsphereReplicationService_SvcThread-14697, T:'Replication Group Creator On VRS', R:protect-8214199d-###-####, J:0d007955-###-####, S:CREATE_GROUP, , TxId: da5f3cc6-###-####] ERROR c.v.v.h.s.v.j.VRSGroupMappingCreator- Error while
  execution ...
  java.lang.RuntimeException: Error Creating Group
  <timestamps> UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- Received a Failed Job, jobType: VsphereReplicationJobs workflow type: VRSGroupMappingCreator

  
  
  
• HCX IX appliance transport tunnel will remain up, but the appliance will show "System state is critical".
  Below messages can also be noticed :
  "Partition root has problem to create new files. Err: open \/fscheck####: read-only file system",
"Partition log has problem to create new files. Err: open \/var\/log\/fscheck####: read-only file system"
  
  
• HCX IX Appliance on VM Web Console OR Remote Console shows below messages :
  You are in emergency mode
Remounting filesystem read-only



Environment

VMware HCX 4.11 or earlier

Cause

The IX VM guest OS entered a read-only state, resulting in the failure of all migrations/DR tasks.

Resolution

• Reboot the affected IX VM. 
• If the issue is not fixed after reboot, please open a support case with Broadcom Support and refer to this KB article.
  For more information, see Creating and managing Broadcom support cases.
• Also refer Recover HCX Manager and Fleet Appliances after datastore failure.
  
  NOTE: For Fleet Appliances(IX), you can do a redeploy. Refer : Redeploy Service Mesh Appliances
  
  
  

Additional Information

Recover HCX Manager and Fleet Appliances after datastore failure