NSX Upgrade fails at Certificate Validity Checks stage with error "signed fields invalid"
search cancel

NSX Upgrade fails at Certificate Validity Checks stage with error "signed fields invalid"

book

Article ID: 375105

calendar_today

Updated On:

Products

VMware SDDC Manager VMware NSX

Issue/Introduction

  • This can happen during the upgrade of an NSX instance in a VCF 5.2 environment - the error in the SDDC UI lists a problematic certificate ID or problematic certificate IDs and also one or more associated  NSX Managers:

  • The error string "signed fields invalid" is included in the SDDC error output: the certificate or certificates are identified by a long alphanumeric string e.g. 123#####-123e-####-hs####21

Sample lcm logging:

2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,9###62a5####452,b9##] [c.v.v.c.t.i.v.VvsControllerTranslator,http-nio-127.0.0.1-7400-exec-7] VMware Compatibility Service controller threw exception: Failed to generate valid VMwareProductVers
ion using releaseApiVersion and gaBuildNumber fields from VVS release Release(major=8, medium=14, minor=1, productId=88, productName=VMware Aria Operations for Logs, releaseId=17891, update=0, version=8.14.1, compatible=true, gaBuildNumb
er=22806512, releaseApiVersion=null, interops=null, upgrades=null).
2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,9###62a5####452,b9##] [c.v.v.c.t.i.v.VvsControllerTranslator,http-nio-127.0.0.1-7400-exec-7] Skipping unknown version for releaseId 17891
2024-08-09T13:43:21.989+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=###f3b94-####-49c9-####-####e623####,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=nsx-sample.com:_ParallelClusterUpgradeElement,bundleElementId=3793e
####-0dc5-####-a19e-###ae4a0####] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-9] Setting Upgrade Error for stage NSX_UPGRADE_STAGE_MANAGER_PRECHECK, error description Certificate Validity Checks: [The certificate with id 123#####-123e-####-hs####21 failed to parse with error: signed fields invalid. Please delete (if unused) or replace this certificate prior to upgrading., The certificate with id 123#####-123e-####-hs####21 failed to parse with error: si
gned fields invalid. Please delete (if unused) or replace this certificate prior to upgrading.]: nsx1-sample.com, nsx2-sample.com, nsx3-sample.com, remediation NSX upgrade failed in precheck for manager cluster. Check for errors in the LCM
log files at 127.0.0.1:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.

NOTE: NSX utilizes many internal certificates (e.g. Client Auth certificates). To identify the certificate or certificates in question, use the certificate identifiers in the error output to trace the bad certificates in the NSXT UI.

  • The issue can also be seen while running pre-checks for a standalone NSX environment upgrade to 4.2.x

Environment

VMware Cloud Foundation 5.2

NSX 4.1.2.1

Cause

  • This issue is caused by a malformed certificate that contains the certificate Private Key in PEM format in the certificate chain.

e.g.

-----BEGIN CERTIFICATE-----

XXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXX

-----END CERTIFICATE-----

-----BEGIN PRIVATE KEY-----

XXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXX

-----END PRIVATE KEY-----

  • A certificate created like this can be a serious security risk as Private Keys should never be recorded in external or internal facing certificates.
  • As of now, it is suspected that certificates like this can be created due to user error.

Resolution

  • If the certificate(s) are NOT in use, it is safe to delete them from NSX UI. Please refer to the relevant NSX documentation for this purpose.
  • If the certificate(s) ARE in use, they must be REPLACED with new and properly constructed certificates:

  1. In the NSXT UI generate a new self-signed certificate (with 'Service Certificate: NO') and make an API call like below:

        POST https://<nsx-mgr>/api/v1/trust-management/principal-identities?action=update_certificate
        {
            "principal_identity_id": "<PI-id>",
            "certificate_id" : "<new-cert-id>"
        }

  2. Fill in the fields between angled brackets with the appropriate values. 

    The new-cert-id should be obvious once you import/generate the certificate in NSX Manager. The PI-id should already be in the system. If you make the API call:

    GET https://<nsx-mgr>/api/v1/trust-management/principal-identities

    You'll get a list of PIs. One of those will have a certificate_id field containing the certificate that is currently faulty. So you need the 'id' field of that PI for the <PI-id> value.

  3.  Once you have created the cert and made the API call the problematic cert should become free to delete. Delete it.
  4. Then do 'Import Certificate' in the UI (with 'Service Certificate: NO') and put just the -----BEGIN CERTIFICATE-----\n<base-64-pem>\n-----END CERTIFICATE---- data in the 'Certificate Contents' box. Leave the 'Private Key' box empty

  5. Now make another API call:

        POST https://<nsx-mgr>/api/v1/trust-management/principal-identities?action=update_certificate
        {
            "principal_identity_id": "<PI-id>",
            "certificate_id" : "<new-cert-id>"
        }

  6. Make sure the certificate ID is the ID of the certificate you just imported.
  7. Delete the unused self-signed certs you just created.
Powered by Wolken Software