The upgrade of an NSXT instance in a VCF 5.2 environment fails at Certificate Validity Checks stage with error "signed fields invalid"

Products

VMware SDDC Manager VMware NSX-T Data Center

Issue/Introduction

The error in the SDDC UI lists a problematic certificate ID or problematic certificate IDs and also one or more associated NSXT Managers:

The error string "signed fields invalid" is included in the SDDC error output.

The certificate or certificates are identified by a long alphanumeric string e.g. 1234abcd-123e-567f-dcba4321

Sample lcm logging:

2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,996562a5cd71452d,b9ac] [c.v.v.c.t.i.v.VvsControllerTranslator,http-nio-127.0.0.1-7400-exec-7] VMware Compatibility Service controller threw exception: Failed to generate valid VMwareProductVers
ion using releaseApiVersion and gaBuildNumber fields from VVS release Release(major=8, medium=14, minor=1, productId=88, productName=VMware Aria Operations for Logs, releaseId=17891, update=0, version=8.14.1, compatible=true, gaBuildNumb
er=22806512, releaseApiVersion=null, interops=null, upgrades=null).
2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,996562a5cd71452d,b9ac] [c.v.v.c.t.i.v.VvsControllerTranslator,http-nio-127.0.0.1-7400-exec-7] Skipping unknown version for releaseId 17891
2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=51ff3b94-f34e-49c9-ad86-5d40e6230157,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=nsx-sample.com:_ParallelClusterUpgradeElement,bundleElementId=3793e
260-0dc5-41d7-a19e-85dae4a0bf8f] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-9] Setting Upgrade Error for stage NSX_UPGRADE_STAGE_MANAGER_PRECHECK, error description Certificate Validity Checks: [The certificate with id 1234abcd-123e-565
7-hsja4321 failed to parse with error: signed fields invalid. Please delete (if unused) or replace this certificate prior to upgrading., The certificate with id 1234abcd-123e-5657-hsja4321 failed to parse with error: si
gned fields invalid. Please delete (if unused) or replace this certificate prior to upgrading.]: nsx1-sample.com, nsx2-sample.com, nsx3-sample.com, remediation NSX upgrade failed in precheck for manager cluster. Check for errors in the LCM
log files at 127.0.0.1:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.

NOTE: NSXT utilises many internal certificates (e.g. Client Auth certificates). To identify the certificate or certificates in question, use the certificate identifiers in the error output to trace the bad certificates in the NSXT UI.

Environment

VMware Cloud Foundation 5.2

NSXT 4.1.2.1.0

Cause

This issue is caused by a malformed certificate that contains the certificate Private Key in PEM format in the certificate chain.

e.g.

-----BEGIN CERTIFICATE-----

XXXXXXXXXXXXXXXXX

-----END CERTIFICATE-----

-----BEGIN PRIVATE KEY-----

XXXXXXXXXXXXXXXX

-----END PRIVATE KEY-----

A certificate created like this can be a serious security risk as Private Keys should never be recorded in external or internal facing certificates.

As of now, it is suspected that certificates like this can be created due to user error.

Resolution

If the certificate(s) are NOT in use, it is safe to delete them from NSXT. Please refer to the relevant NSXT documentation to do so.

If the certificate(s) ARE in use, they must be REPLACED with new, properly constructed certificates:

In the NSXT UI generate a new self-signed certificate (with 'Service Certificate: NO') and make an API call like below:

POST https://<nsx-mgr>/api/v1/trust-management/principal-identities?action=update_certificate
{
"principal_identity_id": "<PI-id>",
"certificate_id" : "<new-cert-id>"
}
Fill in the fields between angled brackets with the appropriate values.

The new-cert-id should be obvious once you import/generate the certificate in NSX Manager. The PI-id should already be in the system. If you make the API call:

GET https://<nsx-mgr>/api/v1/trust-management/principal-identities

You'll get a list of PIs. One of those will have a certificate_id field containing the certificate that is currently faulty. So you need the 'id' field of that PI for the <PI-id> value.
Once you have created the cert and made the API call the problematic cert should become free to delete. Delete it.
Then do 'Import Certificate' in the UI (with 'Service Certificate: NO') and put just the -----BEGIN CERTIFICATE-----\n<base-64-pem>\n-----END CERTIFICATE---- data in the 'Certificate Contents' box. Leave the 'Private Key' box empty
Now make another API call:

POST https://<nsx-mgr>/api/v1/trust-management/principal-identities?action=update_certificate
{
"principal_identity_id": "<PI-id>",
"certificate_id" : "<new-cert-id>"
}
Make sure the certificate ID is the ID of the certificate you just imported.
Delete the unused self-signed certs you just created.