Error: "transport failure reason: TLS_error" when loading Lifecycle tab in VCF 9.1

Products

VMware Cloud Foundation

Issue/Introduction

The following errors are observed in the VCF Operations UI under Lifecycle:

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

Unable to retrieve fleet lifecycle component details

A similar error is observed in the vcf-fleet-build-service-fleetbuild log:

Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on YYYYMMDDHHMMSSZ
        at org.bouncycastle.jcajce.provider.RFC3280CertPathUtilities.processCertA(Unknown Source)
        at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi_8.engineValidate(Unknown Source)
        at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi_8.build(Unknown Source)
        ... 66 common frames omitted
Caused by: java.security.cert.CertificateExpiredException: certificate expired on YYYYMMDDHHMMSSZ
        at org.bouncycastle.jcajce.provider.X509CertificateObject.checkValidity(Unknown Source) ... 69 common frames omitted YYYY-MM-DDTHH:MM:SS.###Z ERROR fleet-upgrade-service 52 [lcm@4413 scPath="ops" thread="system-task-worker-32" logger="c.b.v.l.f.u.s.i.UpgradePlanSyncServiceImpl" method="sync:115"] [] - Failed to refresh inventory org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FLEET_BUILD_SERVICE_HOSTNAME>:9133/fleet-lcm/v1/sddc-lcms": certificate_unknown(46)

Attempting any Actions on VCF Components may fail with the error "Failed to start task (error may be truncated): Error while making HTTP call:". For example, Stop service or Generate support bundle actions on Identity Broker:

Similar errors are observed in the vcf-sddc-build-service-sddcbuild log:

stdout F com.broadcom.vcf.lcm.task.orchestrator.sdk.exception.StepExecutionException: Error while making the HTTP call:
stdout F Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:9134/sddc-lcm/v1/components/########-####-####-############/support-bundles": certificate_unknown(46) 
stdout F Caused by: java.security.cert.CertificateException: Unable to construct a valid chain   
stdout F Caused by: java.security.cert.CertPathBuilderException: Certification path could not be validated.
stdout F Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on ##############GMT+00:00c

When configuring Backup & Restore under Build > Lifecycle > VCF Management, the task failed with the following error:

Task with ID '#######-####-####-####-############' from SDDC lifecycle with ID '########-####-####-####-############' has completed with failure(s). Check the errors in the next sub-task(s) for more details.

Environment

VMware Cloud Foundation 9.x

Cause

The following services vcf-sddc-lcm, vcf-fleet-lcm, VCF Services Runtime support bundle generator, and salt-raas continue to serve an expired certificate after an automated certificate rotation and require a manual service restart to clear the cached expired certificate.

Example Scenario

Day 0	Cert A is created with a 90-day lifespan
Day 60	Cert B is generated by cert-manager. Cert A still has 30 days left before expiring.
Day 90	Cert A expires. The service fails because it is still holding Cert A in memory. Cert B now has 60 days remaining.
Day 95	User runs the script 5 days after Cert A expired. The restart forces the service to load Cert B, leaving 55 days before Cert B expires.

Resolution

This issue is fixed in VCF 9.1.1

See Download Broadcom products, patches and software to download this release.

VMware Cloud Foundation 9.1.1.0 Release Notes

Temporary Workaround

Follow the steps below to restore functionality immediately. The provided script is a temporary workaround, not a permanent fix. Running it forces the service to reload the currently valid certificate, which gives up to 60 days before the same issue recurs.

Note: How long the workaround lasts depends on when the certificate expired. To determine when the certificate expired see the Certificate Identification section.

Prerequisites

The [email protected] password (Note: in most cases, the password is the same as the vmware-system-user password. If needed, follow KB Resetting the [email protected] password for VMware Cloud Foundation Services Runtime to reset the password.)
The VCF Services runtime FQDN (Note: If VCF Services Runtime FQDN is unknown see the additional information section)

Steps

Download the restart_vsp_services.sh script attached to the KB.
Copy the restart_vsp_services.sh script to the /home/vcf/ directory on the SDDC manager.
SSH into the SDDC Manager with the vcf user and then elevate to root with su.
Set the executable permissions on the restart_vsp_services.sh script:
```
chmod +x restart_vsp_services.sh
```

Execute the script:

./restart_vsp_services.sh --runtime-fqdn <###.###.###>

Certificate Identification

After restarting the services, check the managed certificates’ expiration dates and retry the previously failing operation:

Identify a Control Plane node from the VCF Operations UI.

VCF Operations UI > Build > Lifecycle > Components > VCF services runtime > Nodes

SSH to the Control Plane node with the vmware-system-user.
Elevate to root:
```
sudo -i
```

Run the following commands to check the validity of the certificate:

kubectl get certificate fleet-upgrade-service-intra-cert -n vcf-fleet-lcm -o jsonpath='{.status.notAfter}'
kubectl get certificate sddc-upgrade-service-intra-cert -n vcf-sddc-lcm -o jsonpath='{.status.notAfter}'
kubectl get certificate raas-instance-cert -n salt-raas -o jsonpath='{.status.notAfter}'
kubectl get certificate support-bundle-generator -n vmsp-platform -o jsonpath='{.status.notAfter}'

Additional Information

Identify the VCF Service Runtime FQDN:

SSH into the SDDC Manager with vcf user and then elevate to root with su.

Run the following postgres query:

psql -h localhost -U postgres -d platform -c "select primary_fqdn from vsp_cluster where type='MANAGEMENT';"

This issue is also impacting VCF Automation, refer to KB Error: "transport failure reason: TLS_error" in VCF Automation UI.

If running the restart_vsp_services.sh script results in ERROR: 'Queued' for the tasks, refer to Error status with tasks "Queued" after running restart_vsp_services.sh.

ERROR: Task '#######-####-####-####-############' [restart support-bundle] returned an unrecognized status: 'Queued'
Found component: ID = #######-####-####-####-############
Restarting service 'vcf-sddc-build-service-sddcbuild' on component '#######-####-####-####-############' (vcf-sddc-lcm)…
Restart task submitted: #######-####-####-####-############
Polling task  #######-####-####-####-############
   [0s] status: Pending
   [10s] status: Pending
   [20s] status: Queued

Attachments

restart_vsp_services.sh get_app