HCX Manager may be misconfigured when multiple HCX Managers are deployed or imported at the same time from VCF OPS

Products

VMware HCX

Issue/Introduction

When two or more HCX Managers are deployed, or an existing HCX Manager is imported, at the same time (concurrently) using VCF OPS, one or more of the HCX Managers involved may end up with incorrect or incomplete configuration.

This can result in symptoms such as:

HCX Manager precheck or upgrade operations failing.
Errors indicating that the service account used by SDDC Manager does not have the required roles or permissions on the affected HCX Manager.
vCenter Server, NSX, or Single Sign-On (SSO) settings on the affected HCX Manager not matching what was intended for that site.

Environment

VCF Operations HCX 9.1.1

Cause

This issue only occurs when multiple HCX Manager deployment and/or import operations are triggered concurrently from VCF OPS. It does not occur when HCX Manager operations are performed one at a time.

Resolution

Workaround:

Each HCX Manager retains its own configuration independently. If this issue is encountered, the configuration can be reviewed and corrected directly on the affected HCX Manager using its appliance management interface (API/UI).

Log in to the affected HCX Manager directly using its Appliance Management UI (not through SDDC Manager).
Verify the following configuration areas against what is expected for that specific HCX Manager and site, and correct any that are missing, incorrect, or appear to belong to a different site:
- vCenter registration.
- NSX Manager registration.
- Single Sign-On configuration.
- Add below user groups in HCX Role Mapping. (Refer : https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/workload-mobility/vmware-hcx-user-guide-vcf-9-0/installing-the-hcx-manager-appliance/activate-and-configure-hcx-cloud-manager/configure-role-mapping.html)
  - User Group "domain\Migration Administrators" in vcenter to "Migration User" role in HCX Manager
  - User Group "domain\Migration Users" in vcenter to "Migration User" role in HCX Manager
  - User Group "domain\Migration Appliance Administrators" in vcenter to "Migration Appliance Admin" role in HCX Manager
- SDDC Manager registration — see Correcting SDDC Manager Registration below, as this setting is not available from the HCX Manager UI.

Correcting SDDC manager Registration:

The setting that associates an HCX Manager with its SDDC Manager instance, cannot be viewed or changed from the HCX Manager UI. If this setting is found to be incorrect on an affected HCX Manager, it must be corrected using the HCX Manager REST API, as shown below.

You will need the following values before starting:

HCX Manager FQDN, admin username/password	The affected HCX Manager
Domain ID	Retrieve from SDDC Manager: GET https://<sddc-manager-fqdn>/v1/domains (use the id of the relevant management/workload domain)
SDDC Manager ID and FQDN	Retrieve from SDDC Manager: GET https://<sddc-manager-fqdn>/v1/sddc-managers (use the id and fqdn fields)

Step-1 : Logging into the HCX Manager REST API and obtain a session token

curl -sk -D - -o /dev/null -X POST \
  "https://<hcx-fqdn>/api/admin/v1/sessions" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"username":"<hcx-admin-username>","password":"<hcx-admin-password>","authType":"local"}'

Copy the value of the x-hm-authorization header from the response — this is the session token used in the next steps (referred to below as <token>).

Step-2 : Set the SDDC Manager Registration details

curl -sk -X POST \
  "https://<hcx-fqdn>/api/admin/global/config/sddcm" \
  -H "x-hm-authorization: <token>" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{"data":{"items":[{"config":{"domainId":"<domain-id>","id":"<sddc-manager-id>","url":"<sddc-manager-fqdn>"}}]}}'

A successful response returns HTTP 200 OK for each call.

Additional Information

Until this issue is resolved in a future release, avoid triggering deployment or import of multiple HCX Managers at the same time. Perform HCX Manager deployment and import operations one at a time to avoid encountering this issue.