Gangway pod stuck in ContainerCreating state with error "MountVolume.SetUp failed for volume dex-ca"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

While setting up gangway deployment in TKG the gangway pod can enter a ContainerCreating state and the startup hangs with the following error message:

kubectl describe pods gangway-788758cd8c-fw2xj -n tanzu-system-auth

Events:
  Type     Reason       Age                 From                                     Message
  ----     ------       ----                ----                                     -------
  Normal   Scheduled    2m6s                default-scheduler                        Successfully assigned tanzu-system-auth/gangway-788758cd8c-fw2xj to workload-md-0-554858bd65-fmxgh
Warning  FailedMount  62s (x8 over 2m6s)  kubelet, workload-md-0-554858bd65-fmxgh  MountVolume.SetUp failed for volume "dex-ca" : configmap "dex-ca" not found

This error is caused due to missing dex-ca ConfigMap that is mounted as a volume under path /etc/dex/ in the gangway container.

Resolution

This error can be fixed by retrieving the dex-ca.crt from the management cluster and creating a ConfigMap using the contents of dex-ca.crt.

Switch context to management cluster to get the CA for dex and then run the following command:

kubectl get secret dex-cert-tls -n tanzu-system-auth -o 'go-template={{ index .data "ca.crt" }}' | base64 -D > dex-ca.crt

Switch context to workload cluster where you have configured gangway and create a ConfigMap using the dex-ca.crt file from the previous step.

kubectl create cm dex-ca -n tanzu-system-auth --from-file=dex-ca.crt=dex-ca.crt