The renewed CA certificate don't get updated in the CA section of Envoy and Contour certificate's secrets

Products

• VMware Tanzu Application Platform

Issue/Introduction

There is a known issue in earlier TAP (Tanzu Application Platform) versions that the renewed CA certificate don't get updated in the CA section of Envoy and Contour certificate's secrets.

As a symptom, you will see the contour package enters Reconcile failed status. 

$ kubectl get app -n tap-install

NAME                       DESCRIPTION                                                                       SINCE-DEPLOY   AGE
......
contour                    Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)   6m2s           497d

Below error message will also be observed in the Envoy pod log.

[2024-05-31 05:44:02.334][1][warning][config] [./source/common/config/grpc_stream.h:191] StreamClusters gRPC config stream to contour closed since 7510s ago: 14, upstream connect error　or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Cause

This is due to a known issue that the renewed CA certificate don't get updated in the CA section of Envoy and Contour certificate's secrets. 

Resolution

• The temporary workaround:

Please follow the steps in the KB - https://knowledge.broadcom.com/external/article?legacyId=90811 to delete the envoy and contour secret to generate new secrets from most recent CA

kubectl delete secret -n tanzu-system-ingress contourcert
kubectl delete secret -n tanzu-system-ingress envoycert

• The permanent fix:

This known issue has been fixed in the TAP v1.12. Therefore, please schedule to upgrade to the fix version to mitigate the issue.