VCF 9.1 upgrade fails at NSX planning phase

Products

• VMware SDDC Manager / VCF Installer
• VCF Operations

Issue/Introduction

• In the SDDC Manager UI, the VMware Cloud Foundation (VCF) 9.1 upgrade precheck fails during the NSX planning phase with the following error: Resource validations timed out.
  
  
  
  
• Review of the SDDC Manager logs reveals certificate validation and SSL handshake failures.
• Under SDDC Manager's /var/log/vmware/vcf/domainmanager/domainmanager.log, below error log snips are found:
  
  YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,############,####] [c.v.v.q.i.p.n.NsxVersionDataProvider,http-nio-127.0.0.1-####-exec-10]  Sends the following request to '<NSX-T MANAGER FQDN>': com.vmware.nsx.node.Version
  YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,############,####] [c.v.v.s.t.DynamicTrustManager,VLSI-I/O reactor-0]  Error checking certificate chain C=<COUNTRY>, ST=<STATE>, L=<LOCALITY>, O=VMware Inc., OU=<OU>, CN=cluster-<NSX-T MANAGER FQDN>, SerialNumber=############ for validity.
  java.security.cert.CertificateException: Unable to construct a valid chain
          at org.bouncycastle.jsse.provider.ProvX509TrustManager.validateChain(ProvX509TrustManager.java:321)
          at java.base/java.lang.Thread.run(Thread.java:1583)
  Caused by: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
          at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi_8.engineBuild(Unknown Source)
  
  

• Under SDDC Manager's /var/log/vmware/vcf/operationsmanager/operationsmanager.log, below error log snips are found:

  YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,############,####]] [c.v.e.s.c.c.CertificateRetrustService,om-scheduler-1] Failed to retrust with the server https://<NSX-T MANAGER FQDN>/certificate-management/certificate-bundle, error Unexpected character ('<' (code 60)): expected a valid value (JSON
  String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 29, column: 1]
  com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 29, column: 1]
          at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2672)
          at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:742)
  YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,############,####]]] [c.v.v.w.service.WatermarkService,om-scheduler-1] Error while calling create Watermark api for the NSX Manager:
  com.vmware.vapi.client.exception.SslException: certificate_unknown(46)
          at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:107)
          at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:52)
          at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:74)
          at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:288)

Environment

• VCF Operations Manager
• SDDC Manager
• NSX Manager
• VMware Cloud Foundation 9.1

Cause

The SDDC Manager database contains an outdated or un-trusted certificate chain for the component resource (NSX Manager). This typically occurs when a certificate is renewed, updated, or imported directly on the resource node (e.g., via the NSX Manager UI or CLI) instead of being managed through the SDDC Manager orchestration interface. As a result, SDDC Manager loses trust in the endpoint, leading to validation timeouts during the upgrade planning phase.

Resolution

To resolve this issue, manually synchronize and trust the active certificate chain within SDDC Manager:

1. Log in to the SDDC Manager UI using Single Sign-On (SSO) credentials.
2. From the left navigation pane, navigate to Inventory > Workload Domains.
3. Select the Workload Domain associated with the failure.
4. Click on the Certificates tab.
5. Locate the NSX Manager Certificate and review its validation status.
6. Select the option to review the endpoint certificate.
   
   
   
   
7. Verify the certificate details and click Trust to manually import and synchronize the active chain into the SDDC Manager database.
8. Once the certificate status shows as Healthy, navigate back to the upgrade dashboard.
9. Click Retry on the failed precheck task to proceed with the VCF 9.1 upgrade.