VMware NSX Federation LOCAL_MANAGER or GLOBAL_MANAGER replaced expired certificate still alerting as expired

Products

VMware NSX

Issue/Introduction

Symptoms:

You are using Federation.
You had Federation since 3.1.x with a Global manager certificate and upgraded to 3.2.x.
You have an alert that LOCAL_MANAGER or GLOBAL_MANAGER certificates have expired.
You followed the steps in the VMware NSX administration guide step 6. to replace the Federation PI certificates:
You still see alerts in the UI (System - Certificates) indicating there are expired certificates and they still have 'Where Used' count greater than 0.
You are unable to remove the expired certificate, as the system is reporting that it is still in use.
Running the following API call, we see the certificate which is alerting in the UI (take note of the certificate UUID from the UI) is of type: "service_types" : [ "CLIENT_AUTH" ] with no GLOBAL_MANAGER or LOCAL_MANAGER entry co-located on that certificate.

GET api/v1/trust-management/certificates/<certificate_UUID>
{
"pem_encoded" : "-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----",
"has_private_key" : false,
"used_by" : [ {
"node_id" : "{name: 'localmanageridentity',node_id: '<node_using_certificate_UUID>',certificate_id: '<certificate_UUID_from_UI>'}",
"service_types" : [ "CLIENT_AUTH" ]
} ],
"resource_type" : "certificate_self_signed",
"id" : "bbf6a54f-####-####-####-296f3e6634d4",
"display_name" : "<certificate_UUID_from_UI>",
...
}

Checking the Principle Identities (PI) using the following API we see they are now using the new certificate provided in the replacement API call from VMware NSX administration guide step 6 and not the stale cert identified in the previous step.
- For more info please see Principal Identity (PI) Users for NSX Federation .

GET api/v1/trust-management/principal-identities

Note: The presence of "service_types": ["CLIENT_AUTH"] on a certificate alone is not sufficient to indicate a match to this issue.

CLIENT_AUTH service_type is indicative of a Principal Identity (PI) cert, but is not necessarily indicative of a Federation PI certificate. If Principal Identity certs (but not Federation PI certs) are needing to be updated, please see the VMware NSX Administration guide PI section .

A Federation PI cert is characterized by either one of the following:

Presence of CLIENT_AUTH and LOCAL_MANAGER (or GLOBAL_MANAGER) service_types on a single certificate with "node_id": containing name: 'localmanageridentity' or name: 'globalmanageridentity'.
- These entries have been noted to be stale when confirmed with all of the symptoms above this note are present.
Presence of LOCAL_MANAGER or GLOBAL_MANAGER service_type alone.
- This is the certificate entry on the cluster/site that holds the private key for this cert and these entries have not been noted to be stale and should not be released.

Environment

VMware NSX-T

Cause

In a Federation environment, each manager will generate its own certificate which is used for communications.
Each manager will send this certificate to the other sites in the environment, which will then use that certificate for the Principle Identity which is used to configure the other managers.
When we replace the expired certificate, the other managers should replace the certificate in use by the PI that is used to communicate with the manager, which renewed its certificate.
This is an automatic process carried out when the certificate is renewed.
Due to an issue with the automatic release process, there is a stale entry and the manager believes the certificate is still being held by one of the other managers.
This occurs in Federation environments that where deployed in 3.1 or prior and upgraded to 3.2.x.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

Should you identify a stale Federation Principal Identity (PI) certificate according to the guidelines from this article. Please open a Service Request with VMware GSS in order to resolve this issue.