Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1

Products

• VMware Cloud Foundation

Issue/Introduction

• The following errors are observed in the VCF Operations UI under Lifecycle:

  upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
  
  Unable to retrieve fleet lifecycle component details

  
  
  
• A similar error is observed in the vcf-fleet-build-service-fleetbuild log.

  Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on 20260713150319GMT+00:00
          at org.bouncycastle.jcajce.provider.RFC3280CertPathUtilities.processCertA(Unknown Source)
          at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi_8.engineValidate(Unknown Source)
          at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi_8.build(Unknown Source)
          ... 66 common frames omitted
  Caused by: java.security.cert.CertificateExpiredException: certificate expired on 20260713150319GMT+00:00
          at org.bouncycastle.jcajce.provider.X509CertificateObject.checkValidity(Unknown Source)
          ... 69 common frames omitted
  2026-07-15T22:58:06.866Z ERROR fleet-upgrade-service 52 [lcm@4413 scPath="ops" thread="system-task-worker-32" logger="c.b.v.l.f.u.s.i.UpgradePlanSyncServiceImpl" method="sync:115"] []  - Failed to refresh inventory
  org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FLEET_BUILD_SERVICE_HOSTNAME>:9133/fleet-lcm/v1/sddc-lcms": certificate_unknown(46)

Environment

VMware Cloud Foundation 9.1

Cause

The following services vcf-sddc-lcm, vcf-fleet-lcm, VCF Services Runtime support bundle generator, and salt-raas continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate. 

Example Scenario

Day 0	Cert A is created with a 90-day lifespan
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

Engineering is aware of this and a permanent fix is scheduled for the upcoming maintenance release.

Temporary Workaround

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section. 

Prerequisites

• The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
  
  
• The VCF Services runtime FQDN (Note: If VCF Services Runtime FQDN is unknown please see the additional information section) 

Steps

1. Download the restart_vsp_services.sh script attached to the KB.
   
   
2. Copy the restart_vsp_services.sh script to the /home/vcf/ directory on the SDDC manager.
   
   
3. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
4. Set the executable permissions on the restart_vsp_services.sh script:

   chmod +x restart_vsp_services.sh

5. Execute the script:

   ./restart_vsp_services.sh --runtime-fqdn <###.###.###>

Certificate Identification

To determine the exact time the certificate was generated follow the steps below:

1. Identify a Control Plane node from the VCF Operations UI.
   
   

   VCF Operations UI > Build > Lifecycle > Components > VCF services runtime > Nodes
   

   
   
   
   
2. SSH to the Control Plane node with the vmware-system-user.
   
   
3. Elevate to root:

   sudo -i

4. Run the following commands to check the validity of the certificate:

   kubectl get certificate fleet-upgrade-service-intra-cert -n vcf-fleet-lcm -o jsonpath='{.status.notAfter}'
   kubectl get certificate sddc-upgrade-service-intra-cert -n vcf-sddc-lcm -o jsonpath='{.status.notAfter}'
   kubectl get certificate raas-instance-cert -n salt-raas -o jsonpath='{.status.notAfter}'
   kubectl get certificate support-bundle-generator -n vmsp-platform -o jsonpath='{.status.notAfter}'

Additional Information

Identify the VCF Service Runtime FQDN.

1. SSH into the SDDC Manager with vcf user and then elevate to root with su. 
   
   
2. Run the following postgres query:
   psql -h localhost -U postgres -d platform -c "select primary_fqdn from vsp_cluster where type='MANAGEMENT';"
   
   

This issue is also impacting VCF Automation; refer to KB Error: "transport failure reason: TLS_error" in VCF Automation UI.