Tanzu Mission Control "Contour" package reconciliation failed after upgrading TKC in vSphere with Tanzu

Products

• Tanzu Mission Control
• VMware Tanzu Mission Control

Issue/Introduction

You will observe installed "Contour" package in FAILED state under Tanzu Mission Control after the cluster upgrade. 

 

You will see the below error in the PackageInstall CR 

kubectl -n tkg-system get pkgi contour -o yaml

 

The contour and envoy pods are running but you will notice "SSLV3_ALERT_BAD_CERTIFICATE"  related warnings in the pod logs :

[1][warning][config] [k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamListeners gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

Environment

VMware Tanzu Kubernetes Grid Service (TKGs)

VMware Tanzu Mission Control (TMC)

Cause

The CA that signed the contour envoy certificate has expired in vSphere with Tanzu guest cluster.

kubectl get secret envoycert -n tanzu-system-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates
kubectl get secret contourcert -n tanzu-system-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

 

Resolution

Delete the envoy and contour secret to generate new secrets from most recent CA

kubectl delete secret -n tanzu-system-ingress contourcert

kubectl delete secret -n tanzu-system-ingress envoycert

 

NOTE : Take a backup of "contourcert" and "envoycert"  secrets before deleting it.