Error: "transport failure reason: TLS_error" in VCF Automation UI

Products

• VCF Automation
• VMware Cloud Foundation

Issue/Introduction

• Users are unable to log in to the VCF Automation UI.
• The following error message appears within the UI:

  upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

  

Environment

• VCF Automation 9.1
• VMware Cloud Foundation 9.1

Cause

Several services on VCF Automation continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate. 

Example Scenario

Day 0	Cert A is created with a 90-day lifespan by cert-manager.
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

Resolution:

This issue is fixed in VCF 9.1.1. Please refer to the VCF Automation 9.1.1.0 Release Notes for complete details. 

Temporary Workaround

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section. 

Prerequisites

• The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
  
  
• The VCF Automation Services runtime FQDN. (If the VCF Automation Services runtime is unknown see the screenshot in the Additional Information)

Steps

1. Download the restart-vcfa-services.sh script attached to the KB.
   
   
2. Copy the restart-vcfa-services.sh script to the /home/vcf/ directory on the SDDC Manager.
   
   
3. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
4. Set the executable permissions on the restart-vcfa-services.sh script:

   chmod +x restart-vcfa-services.sh

5. Execute the script:

   # Login issues
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm
   
   # Specific pages not loading
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --ui
   
   # Support bundle downloads failing TLS validation
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --support-bundle
   
   # Any combination
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm --ui --support-bundle

Certificate Identification

To determine the exact time the certificate was generated, follow the steps below:

1. SSH to the VCF Automation VIP with the vmware-system-user.
   
   
2. Elevate to root:

   sudo -i

3. Run the following commands to check the validity of the certificate:

   kubectl -n prelude get certificate services-internal -o jsonpath='{.status.notAfter}'
   kubectl -n prelude get certificate tenant-manager -o jsonpath='{.status.notAfter}'
   

Additional Information

This issue is also impacting VCF Service Runtime; refer to KB Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1.


Determining the VCF services runtime FQDN for VCF Automation. 

VCF Operations UI > Build > Lifecycle > VCF Management > Components > VCF Automation