NSX-T upgrade pre check never completes

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T is being upgraded from 3.2.x or 4.0.1.x to 4.1
• When the pre checks run, they never complete and they are continually spinning:

• This may occur during MP (Management Plane - NSX-T managers) upgrade phase or at the start of the upgrade, before edge and host are upgraded.
• The following ERROR is seen in the NSX-T manager log: var/log/upgrade-coordinator/logical-migration.log

2023-02-20T17:28:48.098Z INFO netty-11 NettyClientRouter 4744 Connect Async 192.168.1.1:9041
2023-02-20T17:28:48.099Z ERROR netty-11 ClientHandshakeHandler 4744 exceptionCaught: Exception DecoderException caught.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired

• Restarting NSX-T manager does not resolve the issue.
• Running the following command, as root on the NSX-T manager, shows the corfu server certificate has expired:

openssl x509 -noout -text -in /config/cluster-manager/corfu/public/certificate.pem | grep "Not After"

 

Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

The certificate for corfu server (an internal certificate), prior to the upgrade has expired. As part of the upgrade flow, the pre checks carry out a dry run, the dry run fails due to the expired certificate and the pre check never completes.

Resolution

This issue is resolved in VMware NSX 4.1.1

Workaround:

1. If you have already started the pre checks, stop them in the UI, then continue to step 2
2. Run the following command on all three manager nodes:
   • touch /usr/share/corfu/conf/DISABLE_CERT_EXPIRY_CHECK
3. Re run the pre checks again and continue the upgrade.

Note: After the upgrade is complete, there is no need to remove the created file, it can be ignored.
**To confirm if the expiry check was successfully disabled, you can check the upgrade-coordinator.log for the following entry:
var/log/upgrade-coordinator/upgrade-coordinator.log

INFO netty-0 ReloadableTrustManager 3038 Certificate expiry check has been disabled with: /usr/share/corfu/conf/DISABLE_CERT_EXPIRY_CHECK


If the issue is not resolved and you are noticing the 'Stopping Pre checks' (Refer the screenshot below).

Please open a support request with Broadcom Support and refer to this KB article.