Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1

Products

• VMware Cloud Foundation

Issue/Introduction

• The following errors are observed in the VCF Operations UI under Lifecycle:

  upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
  
  Unable to retrieve fleet lifecycle component details

  
  
  
• A similar error is observed in the vcf-fleet-build-service-fleetbuild log.

  Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on YYYYMMDDHHMMSSZ
          at org.bouncycastle.jcajce.provider.RFC3280CertPathUtilities.processCertA(Unknown Source)
          at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi_8.engineValidate(Unknown Source)
          at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi_8.build(Unknown Source)
          ... 66 common frames omitted
  Caused by: java.security.cert.CertificateExpiredException: certificate expired on YYYYMMDDHHMMSSZ
          at org.bouncycastle.jcajce.provider.X509CertificateObject.checkValidity(Unknown Source) ... 69 common frames omitted YYYY-MM-DDTHH:MM:SS.###Z ERROR fleet-upgrade-service 52 [lcm@4413 scPath="ops" thread="system-task-worker-32" logger="c.b.v.l.f.u.s.i.UpgradePlanSyncServiceImpl" method="sync:115"] [] - Failed to refresh inventory org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FLEET_BUILD_SERVICE_HOSTNAME>:9133/fleet-lcm/v1/sddc-lcms": certificate_unknown(46)

Environment

VMware Cloud Foundation 9.1

Cause

The following services vcf-sddc-lcm, vcf-fleet-lcm, VCF Services Runtime support bundle generator, and salt-raas continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate. 

Example Scenario

Day 0	Cert A is created with a 90-day lifespan
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

Resolution

This issue is fixed in VCF 9.1.1 

See Download Broadcom products, patches and software to download this release. 

VMware Cloud Foundation 9.1.1.0 Release Notes

Temporary Workaround

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section. 

Prerequisites

• The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
  
  
• The VCF Services runtime FQDN (Note: If VCF Services Runtime FQDN is unknown see the additional information section) 

Steps

1. Download the restart_vsp_services.sh script attached to the KB.
   
   
2. Copy the restart_vsp_services.sh script to the /home/vcf/ directory on the SDDC manager.
   
   
3. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
4. Set the executable permissions on the restart_vsp_services.sh script:

   chmod +x restart_vsp_services.sh

5. Execute the script:

   ./restart_vsp_services.sh --runtime-fqdn <###.###.###>

Certificate Identification

To determine the exact time the certificate was generated follow the steps below:

1. Identify a Control Plane node from the VCF Operations UI.
   
   

   VCF Operations UI > Build > Lifecycle > Components > VCF services runtime > Nodes
   

   
   
   
   
2. SSH to the Control Plane node with the vmware-system-user.
   
   
3. Elevate to root:

   sudo -i

4. Run the following commands to check the validity of the certificate:

   kubectl get certificate fleet-upgrade-service-intra-cert -n vcf-fleet-lcm -o jsonpath='{.status.notAfter}'
   kubectl get certificate sddc-upgrade-service-intra-cert -n vcf-sddc-lcm -o jsonpath='{.status.notAfter}'
   kubectl get certificate raas-instance-cert -n salt-raas -o jsonpath='{.status.notAfter}'
   kubectl get certificate support-bundle-generator -n vmsp-platform -o jsonpath='{.status.notAfter}'

Additional Information

Identify the VCF Service Runtime FQDN.

1. SSH into the SDDC Manager with vcf user and then elevate to root with su. 
   
   
2. Run the following postgres query:
   psql -h localhost -U postgres -d platform -c "select primary_fqdn from vsp_cluster where type='MANAGEMENT';"
   
   

This issue is also impacting VCF Automation; refer to KB Error: "transport failure reason: TLS_error" in VCF Automation UI.