Error: "transport failure reason: TLS_error" in VCF Automation UI

Products

• VCF Automation
• VMware Cloud Foundation

Issue/Introduction

• Users are unable to log in to the VCF Automation.
• The following error message appears in the UI:

  upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

Environment

• VCF Automation 9.1
• VMware Cloud Foundation 9.1

Cause

Several services on VCF Automation continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate. 

Example Scenario

Day 0	Cert A is created with a 90-day lifespan by cert-manager.
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

This issue is fixed in VCF 9.1.1.

• Reference Link: VCF Automation 9.1.1.0 Release Notes

Temporary Workaround:

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section. 

Prerequisites

• The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
• The VCF Automation Services runtime FQDN. (If the VCF Automation Services runtime is unknown see the screenshot below)
• Determining the VCF services runtime FQDN for VCF Automation.
  

  VCF Operations UI > Build > Lifecycle > VCF Management > Components > VCF Automation

Steps

1. Download the restart-vcfa-services.sh script attached to the KB.
   
   
2. Copy the restart-vcfa-services.sh script to the /home/vcf/ directory on the SDDC Manager.
   
   
3. SSH into the SDDC Manager with the vcf user and then elevate to root with su.
   
   
4. Set the executable permissions on the restart-vcfa-services.sh script:

   chmod +x restart-vcfa-services.sh

5. Execute the script:

   # Login issues
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm
   
   # Specific pages not loading
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --ui
   
   # Support bundle downloads failing TLS validation
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --support-bundle
   
   # Any combination
   ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm --ui --support-bundle

If the above steps do not successfully renew the certificates, contact Broadcom Support referencing this article (#450789)

• Creating and managing Broadcom support cases

Certificate Identification:

To determine the exact time the certificate was generated, follow the steps below:

1. SSH to the VCF Automation VIP with the vmware-system-user.
2. Elevate to root:

   sudo -i

3. Run the following commands to check the validity of the certificate:

   kubectl -n prelude get certificate services-internal -o jsonpath='{.status.notAfter}'
   kubectl -n prelude get certificate tenant-manager -o jsonpath='{.status.notAfter}'

Alternately:

If certificates fail to renew following the steps above, we need to delete the relevant secrets & certificate requests for these services:

1. Delete Tenant Manager secrets and certificate requests:

   kubectl delete secret -n prelude tenant-manager-tls
   kubectl delete certificaterequests -n prelude tenant-manager-1
   kubectl delete certificaterequests -n prelude tenant-manager-2

2. Delete Postgres secrets and certificate requests by following the KB 'Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Automation'.
3. Re-run the restart-vcfa-services.sh script as above, for example:
   • ./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm --ui --support-bundle

Additional Information

Reference KBs:

1. Refer to the KB for "Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Automation"
2. For issues impacting VCF Service Runtime; refer to KB Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1.