Error: "transport failure reason: TLS_error" in VCF Automation UI

Products

• VCF Automation
• VMware Cloud Foundation

Issue/Introduction

• Users are unable to log in to the VCF Automation UI.
• The following error message appears within the UI:

  upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

  

Environment

• VCF Automation 9.1
• VMware Cloud Foundation 9.1

Cause

Several services on VCF Automation continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate. 

Example Scenario

Day 0	Cert A is created with a 90-day lifespan by cert-manager.
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

Engineering is aware of this and a permanent fix is scheduled for the upcoming maintenance release.

Temporary Workaround

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section. 

Prerequisites

• The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
  
  
• The VCF Automation Services runtime FQDN. (If the VCF Automation Services runtime is unknown see the screenshot in the Additional Information)

Steps

1. Download the restart-vcfa-services.sh script attached to the KB.
   
   
2. Copy the restart-vcfa-services.sh script to the /home/vcf/ directory on the SDDC Manager.
   
   
3. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
4. Set the executable permissions on the restart-vcfa-services.sh script:

   chmod +x restart-vcfa-services.sh

5. Execute the script:

   # Login issues
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm
   
   # Specific pages not loading
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --ui
   
   # Support bundle downloads failing TLS validation
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --support-bundle
   
   # Any combination
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm --ui --support-bundle

Certificate Identification

To determine the exact time the certificate was generated, follow the steps below:

1. SSH to the VCF Automation VIP with the vmware-system-user.
   
   
2. Elevate to root:

   sudo -i

3. Run the following commands to check the validity of the certificate:

   kubectl -n prelude get certificate services-internal -o jsonpath='{.status.notAfter}'
   kubectl -n prelude get certificate tenant-manager -o jsonpath='{.status.notAfter}'
   

Additional Information

This issue is also impacting VCF Service Runtime; refer to KB Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1.


Determining the VCF services runtime FQDN for VCF Automation. 

VCF Operations UI > Build > Lifecycle > VCF Management > Components > VCF Automation