Recover HCX Connector or Cloud Manager after datastore failure
search cancel

Recover HCX Connector or Cloud Manager after datastore failure

book

Article ID: 328982

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

Provide a procedure to attempt to recover the HCX Connector or Cloud Manager VM

Symptoms:

  • HCX Connector or Cloud Manager services are unresponsive
  • VM is unable to boot, waiting in Single User Mode prompt
  • VM may still be active on the network and reply to ICMP requests
  • HCX appliance boots into emergency mode

Cause

After a datastore failure, the VM is unable to mount one or more file systems

Resolution

In the event that the recovery process described in the workaround fails, restore VM from backup
If a backup is not available, re-deployment will be necessary.

Workaround:

  • Access VM console through vCenter
  • Single User Mode requires ROOT password
  • Identify unmounted file systems during boot up (i.e. /common )
  • Run fsck on all file systems that failed to mount:
    • fsck -y /common
  • Reboot
  • Verify VM is able to boot
  • Clean up mongoDB (mongoDB is deprecated as of HCX 4.3.x)
    • Access console via vCenter or SSH into VM as "admin"
    • mongod -repair


If the root ( / ) file system is corrupted, the VM should be considered unrecoverable.

Additional Information

Impact/Risks:

  • All HCX management services will be down due to the system not being able to boot.
  • NE appliances will remain operational and the L2C data path will continue to forward traffic. 
  • All migration and configuration workflows will not be serviced.
  • There is no risk in executing the workaround procedure as the VM may be considered unrecoverable already.