Remediating expired certificates with the CARR script fails to replace certificates

Products

• VMware NSX

Issue/Introduction

You are remediating expired or otherwise certificates by following the instructions at KB369034 and observing the following:

• After identifying the certificates to be replaced and applying the fixes, the script's validation phase still lists the same expired or otherwise invalid certificates.
• The replacement certificates are imported and listed as valid in the NSX UI under System > Certificates but remain unused, with a "Where Used" count of zero, while the expired or invalid certificates remain in use

Environment

VMware NSX

Cause

The CARR script relies on global UTC time to generate new certificates which are currently valid. If the system time on the NSX managers is out of sync and set into the future by a large offset, the newly generated replacement certificates do not pass validation when they are getting applied, as a result the CARR script silently fails to apply the fix and no change of certificate happens.

Resolution

Check that the NTP settings on your NSX Manager nodes are correct, and that your NTP source is in sync with other global sources of time.

Correct the source of time in the NTP server so that the NSX managers' system time is in sync, then run the CARR script again.

Workaround:

Alternatively you can disable NTP time synchronization with NSXCLI or from the NSX UI, and manually set the system time in sync from CLI, such as with a command date -s "YYYY-MM-DD hh:mm:ss", before running the CARR script again.

Additional Information

The carr.log file produced during a run of the CARR script logs the action=apply_certificate API call which effects the certificate change, and the following line will show the error returned by the NSX manager as for why it refuses to validate the new certificate despite its validity.

The new certificate, unused and listed in the NSX UI, can be verified manually to confirm its validity, and the reason for its rejection ("Certificate is not valid yet") can be confirmed with an API query:

GET https://<nsx-mgr>/api/v1/trust-management/certificates/<certificate-id>?action=validate