Contour envoy pods fails with SSLV3_ALERT_BAD_CERTIFICATE or CERTIFICATE_VERIFY_FAILED error

Products

• VMware Tanzu Kubernetes Grid
• vSphere with Tanzu

Issue/Introduction

The Contour envoy pods provided warning messages with this command:

kubectl -n tanzu-system-ingress logs daemonset/envoy -c envoy

[1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamListeners gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

or

[1][warning][config] [./source/common/config/grpc_stream.h:201] StreamRuntime gRPC config stream to contour closed since 3210431s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

The envoy and contour certificates are valid. To check certificate validity, use the command:

kubectl get certificates -n tanzu-system-ingress -o wide

Environment

• vSphere with Tanzu
• VMware Tanzu Kubernetes Grid 2.x

Cause

The CA that signed the envoy/contour certificate has expired:

kubectl get secret envoycert -n tanzu-system-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

kubectl get secret contourcert -n tanzu-system-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

Resolution

1. Verify the most recent CA is valid
kubectl get secrets -n tanzu-system-ingress contour-ca-key-pair -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

2. Delete the envoy and contour secret to generate new secrets from the most recent CA
kubectl delete secret -n tanzu-system-ingress contourcert
kubectl delete secret -n tanzu-system-ingress envoycert

3. Check if the new secrets are generated
kubectl get secret -n tanzu-system-ingress

4. Check if the new certs are valid

kubectl get secret -n tanzu-system-ingress contourcert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -serial -dates -noout

kubectl get secret -n tanzu-system-ingress contourcert -o jsonpath='{.data.tls\.crt}'| base64 -d | openssl x509 -serial -dates -noout

kubectl get secret -n tanzu-system-ingress envoycert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -serial -dates -noout

kubectl get secret -n tanzu-system-ingress envoycert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -serial -dates -noout