Standby global manager certificate not in sync with the active global manager in federation environment.

Products

• VMware NSX Networking

Issue/Introduction

This article explains on how to validate the certificate sync between Active Global manager and Standby Global manager / local manager.

The situation is SSL certificate was renewed on standby global manager for components i.e. manager nodes (all 3 manager nodes), Management cluster, Global manager.

The components were mapped to the new certificate and the old certificate had "0" components mapped.

Whenever there is a change in the global manager certificate, the changes are propagated to  other global and local managers which uses the certificate information to communicate with each other.

With this note, On the active global manager and local manager, we could see Manager nodes, Management cluster were showing mapped to new certificate and global manager was still mapped to old certificate.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The certificate update should be synced automatically to all the other nodes. However there could be instance where there are sync issues and the certificates are not synchronized among the global and local managers.

Resolution

Following steps can be performed to identify the certificate used by managers.

- Get the output of the trust management certificates API call from both global and local manager.

GET https://<nsx-mgr>/api/v1/trust-management/certificates

The above API command gives us all the certificates being used by the NSX manager.

- If the certificate was updated on the standby global manager, get the details of the new certificate and search for the certificate in the API call output from Active global manager / local managers.

(Note: certificate UUID will not be same on other managers. It is best to search with the actual certificate itself )


Standby GM Current certificate:
    {
      "_create_time": 1717921222621,
      "_create_user": "admin",
      "_last_modified_time": 1717923582394,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 12,
      "_system_owned": false,
      "display_name": "<Cert_name>",
      "has_private_key": true,
      "id": "<New_cert_ID>",
      "pem_encoded": "-----BEGIN CERTIFICATE-----\nMIID7TCCAtWgAwIBAgIJAOO2ypUQgvbcMA0GCSqGSIb3DQEBCwUAMHExCzAJBgNV\nBAYTAlVTMQswCQYDVQQIEwJDQTESMBAGA1UEBxMJUGFsbyBBbHRvMQ8wDQYDVQQK\nEwZWTXdhcmUxDDAKBgNVBAsTA05TWDEiMCAGA1UEAxMZQVNPS1JOWEdNMzAwLkJU\nTVVBUC5MT0NBTDAeFw0yNDA2MDgxNzQwMjda
.
.
.
myE42K4QP59n\nOc3G/fegwGhKICDHCFnLsHAJDcM9l3e8Q6i5cZiZmXqFoRCGrs2vjL+EJIeSYdtn\nfNaQCWqNvf0MYsBzuLIJcBKe9poSUlgcwid7iMqfPck0Ht2DFuAl4dKZXwBCmPOI\n0x9o+aeOUkI5QeUt/6ivXGwe2NjKPMP5Eww3Q156y8+iaqi16TuRToSFeehlK2TH\nxXxb/VbD+KdFfVvYJV9gtSZ+bopOEtxRBii64/edehXA14YDK+A8rsPFmnJt8TDE\nWg==\n-----END CERTIFICATE-----\n",
      "resource_type": "certificate_self_signed",
      "used_by": [
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "MGMT_CLUSTER",
            "API",
            "GLOBAL_MANAGER"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        }
      ]
    }


Certificate update on active GM 

  {
      "_create_time": 1717921890165,
      "_create_user": "admin",
      "_last_modified_time": 1717922705165,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 5,
      "_system_owned": false,
      "display_name": "<Cert_name>",
      "has_private_key": false,
      "id": "<Cert_ID>",
      "pem_encoded": "-----BEGIN CERTIFICATE-----\nMIID7TCCAtWgAwIBAgIJAOO2ypUQgvbcMA0GCSqGSIb3DQEBCwUAMHExCzAJBgNV\nBAYTAlVTMQswCQYDVQQIEwJDQTESMBAGA1UEBxMJUGFsbyBBbHRvMQ8wDQYDVQQK\nEwZWTXdhcmUxDDAKBgNVBAsTA05TWDEiMCAGA1UEAxMZQVNPS1JOWEdNMzAwLkJU\nTVVBUC5MT0NBTDAeFw0yNDA2MDgxNzQwMjda
.
.
.
myE42K4QP59n\nOc3G/fegwGhKICDHCFnLsHAJDcM9l3e8Q6i5cZiZmXqFoRCGrs2vjL+EJIeSYdtn\nfNaQCWqNvf0MYsBzuLIJcBKe9poSUlgcwid7iMqfPck0Ht2DFuAl4dKZXwBCmPOI\n0x9o+aeOUkI5QeUt/6ivXGwe2NjKPMP5Eww3Q156y8+iaqi16TuRToSFeehlK2TH\nxXxb/VbD+KdFfVvYJV9gtSZ+bopOEtxRBii64/edehXA14YDK+A8rsPFmnJt8TDE\nWg==\n-----END CERTIFICATE-----\n",

      "resource_type": "certificate_self_signed",
      "used_by": [
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "MGMT_CLUSTER",
            "API"
          ]
        },
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "API"
          ]
        }
      ]
    }
 

- From the above snippet, we can see that on Active we can see the new standby certificate associated with all the other services (ex: API, MGMT_Cluster) of Standby GM and only Global Manager service is not associated.

- Global manager cert on Active global-manager for standby GM is  still associated with the old cert ID.

{
      "_create_time": 1628923541763,
      "_create_user": "admin",
      "_last_modified_time": 1628923549017,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 2,
      "_system_owned": false,
      "display_name": "<Cert_ID>",
      "has_private_key": false,
      "id": "q",
      "pem_encoded": "-----BEGIN CERTIFICATE-----


-----END CERTIFICATE-----",
      "resource_type": "certificate_self_signed",
      "tags": [],
      "used_by": [
        {
          "node_id": "<node_UUID>",
          "service_types": [
            "GLOBAL_MANAGER" >>>>>>>>>>>>
          ]
        },
        {
          "node_id": "{name: 'globalmanageridentity',node_id: '<node_UUID>',certificate_id: '<Cert-ID>'}",
          "service_types": [
            "CLIENT_AUTH"
          ]
        }
      ]
    },

- This confirms that the global manager cert is still pointing to the old certificate.

- Check the principal identity certificate on the active global manager. 

- We could see the PI certificate is still pointing to old certificate.

{
        "_create_time": 1628923543264,
        "_create_user": "admin",
        "_last_modified_time": 1628923543264,
        "_last_modified_user": "admin",
        "_protection": "NOT_PROTECTED",
        "_revision": 0,
        "_system_owned": false,
        "certificate_id": "<Cert_ID>", >>> Matches the display name of the old certificate on the Active GM
        "display_name": "<Name of the principal Identity>",
        "id": "<PI_ID>",
        "is_protected": true,
        "name": "GlobalManagerIdentity",
        "node_id": "<node_UUID>", >>> Site ID
        "resource_type": "PrincipalIdentity",
        "role": "enterprise_admin",
        "tags": []
      },


- We can do a manual update of the principal identity certificate on the Active global manager using the below API call.

POST https://<Active_nsx-Global_mgr>/api/v1/trust-management/principal-identities?action=update_certificate
{
    "principal_identity_id": "<PI_ID>",
    "certificate_id" : "<New_cert_ID>"
}


Once this is successful, we can do the same on local managers to update the certificate.