Global Manager fails to import Local Manager configuration due to missing Principal Identity certificate (NSX Federation)
search cancel

Global Manager fails to import Local Manager configuration due to missing Principal Identity certificate (NSX Federation)

book

Article ID: 445754

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • Global Manager (GM) shows "Internal Server Error..." when communicating with a Local Manager (LM).

  • The "Sync Status" in the GM UI Under System > Location Manager > Locations may still show Success with a green dot, but the dot beside the location name is white.

  • The LM UI, under System > Location Manager, shows errors such as "Unable to fetch full sync status" and "Lost communication to <GM Name>".

  • Standard certificate replacement APIs fail:

    • set_pi_certificate_for_federation requires the "old" certificate to still be present.

    • apply_certificate fails because this function doesn't apply to "service_type": "GLOBAL_MANAGER". It can be used to replace a "LOCAL_MANAGER"  certificate

Environment

NSX Federation

Cause

The Principal Identity (PI) on the Local Managers is mapped to a certificate UUID that has been deleted or is missing from the trust store.

Resolution

 To resolve this, you must manually recreate the Principal Identity using the X-Overwrite: true header and assigning a valid certificate as shown in the steps below:

Verification:

  1. Run an API GET to https://<LM_IP>/api/v1/trust-management/principal-identities  and copy the successful response information ("Body").

    • Any LM from the cluster may be used (or a VIP). The LM cluster shares the PI and uses a single certificate per PI. All of the LMs should show have similar, synchronized PI and certificate information.

    • There are typically two GM PIs with one of them being for a Standby Global Manager. There will also be LM PIs for any other sites and PIs for some integrated services where applicable.

    • Each PI has a block of data between brackets beginning with the { before a "name": line and ending with the } right after a "_revision" line.

    • Example output showing just one PI section from the entire JSON body results with a name showing it's a GM PI:
      {
  "name": "GlobalManagerIdentity-########-####-####-####-###########",
  "node_id": "########-####-####-####-###########",
  "role": "enterprise_admin",
  "certificate_id": "########-####-####-####-###########",    <<<only the value from this line gets changed
  "roles_for_paths": [
    {
      "path": "/",
      "roles": [
        {
          "role": "enterprise_admin"
        }
      ],
      "delete_path": false
    }
  ],
  "is_protected": false,
  "resource_type": "PrincipalIdentity",
  "id": "########-####-####-####-###########",     <<<this ID would be used during a DELETE call
  "display_name": "########-####-####-####-###########",
  "_revision": 0
}
  2. Verify the GM ID information by searching the name or node_id in NSX Manager (to know if it's the Active or Standby GM).

  3. Search the certificate_id either as a GET to https://<LM_IP>/api/v1/trust-management/certificates/<certificate_id> or within the Local NSX Manager UI to see if it is indicating:

    • The cert is expired or invalid = different issues that can be resolved with a standard replacement

    • The cert is fully missing: If the search yields no results, the PI is confirmed to point to a certificate that has been deleted. If you identify this issue in your environment, open a Support Request with Broadcom Support. To ensure proper routing and visibility of integration details, the Support Request should be opened under the VMware Cloud Foundation product in the Broadcom Support Portal.

Additional Information

Powered by Wolken Software