HCX-Connector or Cloud Manager restoration from backup

Products

VMware HCX

Issue/Introduction

The process described in this document will restore HCX Connector or Cloud Manager to the state it was in at the time of the backup.

In the event that the HCX Connector or Cloud Manager VM is lost or inadvertently deleted, the content from the HCX database can se restored.

Also, although the upgrade process for HCX deployments is extremely reliable, it is recommended to perform a backup prior to upgrading.

The inability to use the HCX Connector or Cloud Manager will prevent the user from executing migration or configuration workflows yet the most critical Network Extension services will continue to operate without disruption but no configuration changes can be done until the system is restored.

IMPORTANT: The backup file should not be more than two days old otherwise that may also cause data inconsistency between the restored system and any site pairing peers. The content of the backup file supersedes any changes made as part of the upgrade.

Environment

VMware HCX

Resolution

The following steps should be performed to backup the HCX manager before a software upgrade:

Log in to the appliance management interface: <https://hcx-ip-or-fqdn:9443>
Navigate to Administration > Troubleshooting > Backup & Restore.
At the FTP Server Settings section, configure an FTP or SFTP server for uploading the backup file.
After entering the IP/FQDN, Transfer Protocol, Username, Password and Backup directory location, click on SAVE.
Navigate to Backup & Restore > Check the box Upload to server > Click Generate.
The message Backup operation succeeded will indicate that the backup file has been uploaded to the FTP/SFTP server.

There are few prerequisites in order to restore a HCX Connector or Cloud Manager from the latest backup:

Make sure to power OFF the faulty system.
A new system must be deployed with same software version that was running before the upgrade.
The replacement system must have the same IP address, subnet mask, default gateway and DNS server settings.

The following procedure should be performed to restore HCX Connector or Cloud Manager from the latest backup once the new system has been deployed:

Log in to the appliance management interface: <https://hcx-ip-or-fqdn:9443>.
Navigate to Administration > Troubleshooting > Backup & Restore.
At the Backup & Restore section, click on RESTORE to select the backup file.

The restoration process should start. It can take several minutes to complete and the below message will appear:

When the restoration process finishes successfully then below message will appear:

Restart appliance management service upon successful restore operation.

SSH into the Connector or Cloud Manager with "admin" credentials.
Change the user to "root".

# systemctl restart appliance-management

And make sure the appliance-management service STATUS is active

# systemctl status appliance-management
* appliance-management.service - Appliance Management
   Loaded: loaded (/etc/systemd/system/appliance-management.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-12-07 17:41:16 GMT; 27s ago
  Process: 18262 ExecStartPre=/etc/systemd/service-dependency-check.sh ignore-service-failures mongodb zookeeper kafka app-engine web-engin
 Main PID: 18301 (sh)
    Tasks: 19 (limit: 512)

Verify that the system is operating properly:

Navigate to the Dashboard tab and confirm that the NSX and vCenter registration status is green.
Navigate to the Appliance Summary tab and verify that the Hybridity Services, Common Services and System Level Services are running.

Alternatively, the VMware snapshot feature can be used to preserve the state of the system right before a software upgrade. In the unlikely event of a problem after the upgrade, the HCX Connector or Cloud Manager VM can be reverted to the last snapshot.

IMPORTANT: It is strongly recommended for the HCX Connector or Cloud Manager to not have any active migration or configuration workflows while taking a snapshot.
Manager services should be shutdown before taking the snapshot. Those services can be restarted before proceeding with an upgrade.
Refer to HCX User Guide - Monitoring HCX Services for details on how to control the following services:

Application Service
Web Service

NOTE: VMware snapshot feature is not intended to be used as a long-term backup solution for any VM.

Additional Information

KB 025279 vSphere Snapshot Best Practices
KB 86462 Unable to upgrade HCX Connector or Cloud Manager after reverting a snapshot
KB 86397 HCX Connector or Cloud Manager UI inaccessible after restoring a backup