NSX-T upgrade pre check never completes
search cancel

NSX-T upgrade pre check never completes

book

Article ID: 322646

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • NSX-T is being upgraded from 3.2.x or 4.0.1.x to 4.1
  • When the pre checks run, they never complete and they are continually spinning:
pre check spinning.png
  • This may occur during MP (Management Plane - NSX-T managers) upgrade phase or at the start of the upgrade, before edge and host are upgraded.
  • The following ERROR is seen in the NSX-T manager log: var/log/upgrade-coordinator/logical-migration.log
2023-02-20T17:28:48.098Z INFO netty-11 NettyClientRouter 4744 Connect Async 192.168.1.1:9041
2023-02-20T17:28:48.099Z ERROR netty-11 ClientHandshakeHandler 4744 exceptionCaught: Exception DecoderException caught.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
  • Restarting NSX-T manager does not resolve the issue.
  • Running the following command, as root on the NSX-T manager, shows the corfu server certificate has expired:
openssl x509 -noout -text -in /config/cluster-manager/corfu/public/certificate.pem | grep "Not After"
 


Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

The certificate for corfu server (an internal certificate), prior to the upgrade has expired. As part of the upgrade flow, the pre checks carry out a dry run, the dry run fails due to the expired certificate and the pre check never completes.

Resolution

This issue is resolved in VMware NSX 4.1.1, available at VMware downloads.

Workaround:
  1. If you have already started the pre checks, stop them in the UI, then continue to step 2
  2. Run the following command on all three manager nodes:
    • touch /usr/share/corfu/conf/DISABLE_CERT_EXPIRY_CHECK
  3. Re run the pre checks again and continue the upgrade.
Note: After the upgrade is complete, there is no need to remove the created file, it can be ignored.
**To confirm if the expiry check was successfully disabled, you can check the upgrade-coordinator.log for the following entry:
var/log/upgrade-coordinator/upgrade-coordinator.log

INFO netty-0 ReloadableTrustManager 3038 Certificate expiry check has been disabled with: /usr/share/corfu/conf/DISABLE_CERT_EXPIRY_CHECK


If the issue is not resolved and you are noticing the 'Stopping Pre checks' (Refer the screenshot below).

Please open a support request with VMware NSX-T GSS and refer to this KB article.
For more information, see How to Submit a Support Request.

image.png