Recover HCX Connector or Cloud Manager after datastore failure
book
Article ID: 328982
calendar_today
Updated On:
Products
VMware HCX
Issue/Introduction
Provide a procedure to attempt to recover the HCX Connector or Cloud Manager VM
Symptoms:
HCX Connector or Cloud Manager services are unresponsive
VM is unable to boot, waiting in Single User Mode prompt
VM may still be active on the network and reply to ICMP requests
HCX appliance boots into emergency mode
Cause
After a datastore failure, the VM is unable to mount one or more file systems
Resolution
In the event that the recovery process described in the workaround fails, restore VM from backup If a backup is not available, re-deployment will be necessary.
Workaround:
Access VM console through vCenter
Single User Mode requires ROOT password
Identify unmounted file systems during boot up (i.e. /common )
Run fsck on all file systems that failed to mount:
fsck -y /common
Reboot
Verify VM is able to boot
Clean up mongoDB (mongoDB is deprecated as of HCX 4.3.x)
Access console via vCenter or SSH into VM as "admin"
mongod -repair
If the root ( / ) file system is corrupted, the VM should be considered unrecoverable.
Additional Information
Impact/Risks:
All HCX management services will be down due to the system not being able to boot.
NE appliances will remain operational and the L2C data path will continue to forward traffic.
All migration and configuration workflows will not be serviced.
There is no risk in executing the workaround procedure as the VM may be considered unrecoverable already.