Contour envoy pods fails with SSLV3_ALERT_BAD_CERTIFICATE or CERTIFICATE_VERIFY_FAILED error

Products

• VMware Tanzu Kubernetes Grid
• vSphere with Tanzu

Issue/Introduction

The Contour envoy pods fails with::

[1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamListeners gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

or

[1][warning][config] [./source/common/config/grpc_stream.h:201] StreamRuntime gRPC config stream to contour closed since 3210431s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

The envoy and contour certificates are valid. To check certificate validity, use the command:

kubectl get certificates -n tanzu-system-ingress -o wide

Environment

vSphere with Tanzu

VMware Tanzu Kubernetes Grid 1.x

Cause

The CA that signed the envoy certificate has expired:

kubectl get secret envoycert -n tanzu-system-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

Resolution

Verify the most recent CA is valid
kubectl get secrets -n tanzu-system-ingress contour-ca-key-pair -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -text

Delete the envoy and contour secret to generate new secrets from most recent CA
kubectl delete secret -n tanzu-system-ingress contourcert
kubectl delete secret -n tanzu-system-ingress envoycert