Pre-check on SDDC Manager fails on NSX Upgrade from 4.1.x to 9.0.
search cancel

Pre-check on SDDC Manager fails on NSX Upgrade from 4.1.x to 9.0.

book

Article ID: 412099

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • Upgrade NSX from 4.1.x to 9.0
  • In SDDC Manager UI
    • UC NSX Bundle Prechecks step shows:
      error_message: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure reason: delayed connect error: 111, httpStatus:, error_code: 0
    • UC NSX Bundle Prechecks step may also shows:
      NSX Upgrade Coordinator update failed after PUB upload during precheck.
    • NSX UC Rollback after Bundle Precheck step shows:
      UC Rollback Timed out.
  • NSX Upgrade Coordinator log at /var/log/upgrade-coordinator/upgrade-coordinator.log shows:

    <TIMESTAMP>  INFO netty-<ID> ClientHandshakeHandler <ID> channelRead: Removing handshake handler from pipeline.
    <TIMESTAMP> ERROR WrapperStartStopAppMain CorfuRuntime <ID> connect: Couldn't connect to server.
    java.util.concurrent.TimeoutException: null
            at java.util.concurrent.CompletableFuture.timedGet(Unknown Source) ~[?:?]
            at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
            at org.corfudb.runtime.clients.NettyClientRouter.sendRequestAndGetCompletable(NettyClientRouter.java:458) ~[runtime-9.0.20250318191142.8085.1.jar:?]
            at org.corfudb.runtime.clients.AbstractClient.sendRequestWithFuture(AbstractClient.java:43) ~[runtime-9.0.20250318191142.8085.1.jar:?]
            at org.corfudb.runtime.clients.BaseClient.ping(BaseClient.java:51) ~[runtime-9.0.20250318191142.8085.1.jar:?]
            at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
            at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
            at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
            at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]

    <TIMESTAMP>  INFO WrapperStartStopAppMain UpgradeUfoConfig <ID> SYSTEM [nsx@4413 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Got corfudbconnector instance
    <TIMESTAMP>  INFO WrapperStartStopAppMain CorfuRuntime <ID> connect: runtime parameters CorfuRuntime.CorfuRuntimeParameters(maxWriteSize=26214400, bulkReadSize=20, holeFillRetry=10, holeFillRetryThreshold=PT1S, holeFillTimeout=PT10S, mvoCacheExpiry=PT10M, cacheEntryMetricsDisabled=true, cacheDisabled=false, maxCacheEntries=80, maxMvoCacheEntries=50, maxCacheWeight=0, cacheConcurrencyLevel=8, cacheExpiryTime=<CACHE_EXPIRY_TIME>, holeFillingDisabled=false, writeRetry=5, trimRetry=2, checkpointRetries=5, checkpointBatchSize=50, maxUncompressedCpEntrySize=100000000, restoreBatchSize=50, streamBatchSize=10, checkpointReadBatchSize=1, cacheWrites=true, clientName=CorfuClient, checkpointTriggerFreqMillis=0, runtimeGCPeriod=PT20M, disableFileWatcher=false, clusterId=null, systemDownHandlerTriggerLimit=120, layoutServers=[], invalidateRetry=5, priorityLevel=NORMAL, codecType=ZSTD, metricsEnabled=true, highestSequenceNumberBatchSize=4, streamingWorkersThreadPoolSize=2, streamingPollPeriod=PT0.1S, streamingSchedulerPollBatchSize=25, streamingSchedulerPollThreshold=5, sourceCodeVersion=24733065)
    2025-08-27T09:54:20.043Z  INFO netty-1 NettyClientRouter 187934 Connect Async <Manager_IP_ADDRESS>:9000
    2025-08-27T09:54:20.052Z ERROR netty-1 ClientHandshakeHandler 187934 exceptionCaught: Exception DecoderException caught.
    io.netty.handler.codec.DecoderException: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslException: error:0A000438:SSL routines::tlsv1 alert internal error
            at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) ~[netty-handler-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]

  • Corfu on NSX log at /var/log/corfu/corfu.9000.log shows:
    <TIMESTAMP> | INFO  |                       worker-<ID> | o.c.s.t.ReloadableTrustManager | Certificate expiry check has been disabled with: /usr/share/corfu/conf/DISABLE_CERT_EXPIRY_CHECK
    <TIMESTAMP> | DEBUG |                       worker-<ID> | ReferenceCountedOpenSslContext | verification of certificate failed
    sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: TrustAnchor found but certificate validation failed.
            at java.base/sun.security.validator.PKIXValidator.doValidate(Unknown Source)
            at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
            at java.base/sun.security.validator.Validator.validate(Unknown Source)
            at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
            at java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(Unknown Source)
            at org.corfudb.security.tls.ReloadableTrustManager.checkClientTrusted(ReloadableTrustManager.java:41)
            at io.netty.handler.ssl.util.X509TrustManagerWrapper.checkClientTrusted(X509TrustManagerWrapper.java:52)
            at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkClientTrusted(EnhancingX509ExtendedTrustManager.java:62)
            at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslServerContext.java:276)
            at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:797)
            at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:655)
            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1287)
            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1438)
            at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1481)
            at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222)
            at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1443)
            at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
            at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
            at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
            at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
            at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
            at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
            at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
            at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
            at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
            at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
            at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
            at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
            at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
            at java.base/java.lang.Thread.run(Unknown Source)
    Caused by: java.security.cert.CertPathValidatorException: TrustAnchor found but certificate validation failed.
            at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi_8.engineValidate(Unknown Source)
            at java.base/java.security.cert.CertPathValidator.validate(Unknown Source)
            ... 37 common frames omitted
    Caused by: java.security.SignatureException: certificate does not verify with supplied key
            at org.bouncycastle.jcajce.provider.X509CertificateImpl.checkSignature(Unknown Source)
            at org.bouncycastle.jcajce.provider.X509CertificateImpl.verify(Unknown Source)
            at io.netty.handler.ssl.util.LazyX509Certificate.verify(LazyX509Certificate.java:190)
            at org.bouncycastle.jcajce.provider.CertPathValidatorUtilities.verifyX509Certificate(Unknown Source)
            at org.bouncycastle.jcajce.provider.CertPathValidatorUtilities.findTrustAnchor(Unknown Source)
            ... 39 common frames omitted

Environment

NSX 9.0

Cause

Due to upgrade step change in 9.0, Upgrade Coordinator picks incorrect certificate store after upgrade bundle is uploaded.

Resolution

  1. Login on NSX Manager node that shows Upgrade Coordinator Error and run following command.
    mv /opt/vmware/upgrade-coordinator-tomcat/conf/ufo-factory.properties /opt/vmware/upgrade-coordinator-tomcat/conf/ufo-factory.properties.backup 
    cp /opt/vmware/upgrade-coordinator-tomcat/conf/ufo-factory.propertie.bak /opt/vmware/upgrade-coordinator-tomcat/conf/ufo-factory.properties
    systemctl restart upgrade-coordinator
  2. Visit Upgrade page on NSX UI.
  3. Run pre-check on Edge
  4. Start the edge upgrade
  5. Visit SDDC Manager UI to proceed with the remaining components.

NOTE: If you run pre-check on SDDC Manager again, UI may show the same errors.
Follow the step "Login on NSX Manager node that shows Upgrade Coordinator Error and run following command" again.

Powered by Wolken Software