Failure to re-register compute manager due to certificate revocation list (CRL) verification error.

Products

VMware NSX

Issue/Introduction

Unable to re-register compute manager when crl_checking_enabled is set to TRUE

The API below returns a few flags in a JSON structure, one of which is "crl_checking_enabled": "true".

GET https://<NSX Manager>/policy/api/v1/infra/security-global-config

JSON response: 
{
    "crl_checking_enabled": true,
    "ca_signed_only": false,
    "eku_checking_enabled": true,
    "id": "#########-####-####-####-###########",
    "_create_time": 1679339007871,
    "_create_user": "system",
    "_last_modified_time": 1679339007871,
    "_last_modified_user": "system", "_protection": "NOT_PROTECTED",
    "_revision": 0
}

Registration of a compute manager in NSX-T fails with error code 90206.
The error indicates a failure to find the signing principal for the Certificate Revocation List (CRL).

CRL check for certificate of Compute Manager <FQDN_REDACTED> failed because of CRL check failed: Couldn't find signing principal of the CRL CN=<CA_NAME_REDACTED>,O=<ORG_NAME_REDACTED>,C=<REGION_REDACTED>. (Error code: 90206)

NSX & vCenter Handshake analysis via OpenSSL confirms vCenter does not present the full chain of certificate on port 443.
This can be verified using the below command on NSX manager (root mode)

openssl s_client -connect <VC_FQDN>:443 -showcerts

Environment

VMware NSX

Cause

The vCenter Server is not configured to send the full certificate chain during the TLS handshake.
CRL checking, when enabled on the NSX Manager, requires the complete chain to identify the signing authority of the CRL.

Resolution

Append the intermediate CA certificates to the leaf certificate file in the correct signing order on the vCenter MACHINE_SSL_CERT following the below steps on vCenter:

Backup the current machine SSL certificate and private pair.

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > mach.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > mach.key

Delete the existing certificate entry from the MACHINE_SSL_CERT store to allow for a chain update.

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Append the intermediate CA certificates to the leaf certificate file in the correct signing order.
Ensure the root CA is not included in the server certificate chain as per standard TLS requirements.

 Edit mach.crt file and append the intermediate CA certificates in the right order in this fashion.

leaf.cert

intermediate CA1

intermediate CA2

.

.

last intermediate CA

>intermediate CA1 should have signed the leaf and intermediate CA2 should have signed intermediate CA1 and the last >intermediate CA should be signed by the self signed root CA which MUST not be present in the server certificate chain.

Recreate the certificate entry in the VECS store using the updated chain and original private key.

Recreate the Machine SSL Certificate entry in VECS,

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert mach.crt --key mach.key

Restart the rhttpproxy service and all associated services to load the new certificate chain on port 443.

service-control --restart rhttpproxy

service-control --start  --all

Verify the fix by running an OpenSSL connect command to ensure the full chain is presented.

openssl s_client -connect <VC_FQDN>:443 -showcerts

Re-authenticate the vCenter within the NSX Manager interface to complete the registration.

Once you see that the cert chain is getting presented during the TLS between NSX and VC , Please reauth the VC on NSX.

Additional Information

The CRL check functions correctly even if vCenter and NSX certificates are signed by different issuers, provided the chain is intact.