Helm Repo unable to reconcile in TMC 1.4.4

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Helm repositories are failing to reconcile with the error message “x509: certificate signed by unknown authority”. This happens even though certificates are up to date.

Environment

Tanzu Mission Control Self Managed (TMC SM) v1.4.4

Cause

This issue can be caused by network doing TLS inspection. Additional SSL certs get injected, which are not recognized by TMC even if those certs have been added to trusted certs. This can be verified by running the following command after ssh'ing into one of the worker nodes:

openssl s_client -connect raw.githubusercontent.com:443 -showcerts </dev/null

If output shows additional 3rd party certs in chain (besides github.io), these will need to be added to fluxcd-source-controller pod trust store.

Resolution

In TMC SM v1.4.5, issue can be resolved by updating fluxcd-source-controller package to include necessary certs in Package configuration and then restarting the source controller:



If using v1.4.4, the following workaround will need to be applied:

Steps:
1. Download overlay-secret.yaml (attached to KB) and set kubeconfig to the atttached cluster
2. Apply secret

kubectl apply -f overlay-secret.yaml

3. Ensure secret exists
4. Annotate the package install to use this secret

kubectl annotate pkgi source-controller \
  -n tanzu-fluxcd-packageinstalls \
  ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=source-controller-ssl-overlay \
  --overwrite

5. Wait for package to reconcile. After applying the overlay, it'll go in reconcile state and finally reconcile should succeed. (In case it doesnt reconcile automatically, one can kick the package install "kctrl pkg installed kick -i source-controller -n tanzu-fluxcd-packageinstalls", do this only if necessary)

6. Verify the ENV var is set in pod

kubectl get pod -n tanzu-source-controller \
  -l app=source-controller \
  -o jsonpath='{range .spec.containers[?(@.name=="manager")].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep SSL_CERT_FILE

(Expected: SSL_CERT_FILE=/etc/ca-bundle/cert/cert)

7. Check helm repos

kubectl get helmrepository -A

Additional Information

1. Cloudhealth URL has been moved and will be fixed in v1.4.5 release.
2. Bitnami repo will need to be pointed to bitnamilegacy repo if using old images. Keep in mind that bitnamilegacy is archive-only, no updates, no security patches. It's an interim bridge only. Long-term options are either mirroring images to a private registry or subscribing to Bitnami Secure Images. See here for more info.