Recover HCX Manager and Fleet Appliances after datastore failure

Products

• VMware HCX

Issue/Introduction

• HCX Manager and IX/NE may still be active on the network and reply to ICMP requests
• HCX-IX/NE tunnels might be up and running as data path is not impacted.
• HCX  Manager shows "Read-error on swap-device" on VM Web Console OR Remote Console :
  
  
  
  
  
• HCX Fleet Appliances(NE/IX) on VM Web Console OR Remote Console shows below messages :
  
  You are in emergency mode
EXT4-fs error (sda#)
Remounting filesystem read-only
Detected aborted journal
  
  
  
  
  
  
  
  
  

Environment

VMware HCX

Cause

Guest OS filesystems went to read-only mode due to underlying storage issues.

Resolution

If there are storage-related issues, virtual machines (VM) that were previously up and running may start exhibiting unexpected behavior. If the virtual machines do not receive responses from the storage quickly enough, causing the filesystem to go into read-only mode.
In this case, you need to fix the storage issue and then reboot these VMs. 

If the issue is not fixed after reboot, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases


NOTE:
In the event that the recovery process fails, restore VM from backup.
If a backup is not available, re-deployment will be necessary.

Additional Information

• All HCX management services could be down due to the system not being able to boot.
• NE appliances will remain operational and the L2C data path will continue to forward traffic. 
• All migration and configuration workflows will not be serviced.
• There is no risk in executing the workaround procedure as the VM may be considered unrecoverable already.