upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_endSeveral services on VCF Automation continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate.
Example Scenario
| Day 0 | Cert A is created with a 90-day lifespan by cert-manager. |
| Day 60 | Cert B is generated by cert-manager. Cert A still has 30 days left before expiring. |
| Day 90 | Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining. |
| Day 95 | User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires. |
This issue is fixed in VCF 9.1.1.
Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.
[email protected] password for VCFAvmware-system-user password.VCF Operations UI > Build > Lifecycle > VCF Management > Components > VCF Automation
restart-vcfa-services.sh script attached to the KB.restart-vcfa-services.sh script to the /home/vcf/ directory on the SDDC Manager.restart-vcfa-services.sh script:chmod +x restart-vcfa-services.sh# Login issues
./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm
# Specific pages not loading
./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --ui
# Support bundle downloads failing TLS validation
./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --support-bundle
# Any combination
./restart-vcfa-services.sh --runtime-fqdn <RUNTIME_FQDN> --tm --ui --support-bundleIf the above steps do not successfully renew the certificates, contact Broadcom Support referencing this article (#450789)
To determine the exact time the certificate was generated, follow the steps below:
vmware-system-user.sudo -ikubectl -n prelude get certificate services-internal -o jsonpath='{.status.notAfter}'
kubectl -n prelude get certificate tenant-manager -o jsonpath='{.status.notAfter}'
Reference KBs: