TKGi fluent-bit pods failing with "MountVolume.SetUp failed for volume "pks-ca" : secret "pks-ca" not found" and "secret "fluent-bit" not found'

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When using TKGi, it may be observed that the fluent bit pods are in Init:0/1 state. When checking further with kubectl describe pod fluent-bit-xxx -n pks-system , it shows the following in events:

  Warning  FailedMount  6m41s (x3 over 6m42s)  kubelet            MountVolume.SetUp failed for volume "pks-ca" : secret "pks-ca" not found
  Warning  FailedMount  6m39s (x4 over 6m42s)  kubelet            MountVolume.SetUp failed for volume "fluent-bit-certs" : secret "fluent-bit" not found

Environment

TKGi

Cause

As per the errors, the fluent bit pods are failing due to a lack of the pks-ca and fluent-bit secrets. You can validate this by running:

kubectl get secrets -n pks-system

 

Resolution

To resolve this, the missing secrets need to be regenerated. This can be done like so:

1. kubectl get jobs -A
2. This will show a completed job cert-generator-xxx
3. Delete this with kubectl delete job cert-generator-xxx -n pks-system
4. Then delete the observability manager pod like so: kubectl delete pod observability-manager-xxx -n pks-system
5. This will bring back the secrets, which you can validate with kubectl get secrets -n pks-system
6. Finally, restart the fluent bit pods with: kubectl rollout restart daemonset/fluent-bit -n pks-system