Mounting the remote server CA certificate on fluent-bit pods deployed through package in TKG cluster

Products

• VMware Tanzu Kubernetes Grid Plus 1.x

Issue/Introduction

To set TLS ca_cert of elacticsearch in fluent-bit config deployed as package in TKG cluster. One has to pass the cert path in fluent-bit-data-values.yaml file. Also we need to keep the ca_cert on particular path inside pod defined in fluent-bit-data-values.yaml file. This can be done by using overlay-secret to update the running fluent-bit package.

 

Environment

Tanzu Kubernetes Grid v2.x 

Fluent-bit v1.x or v2.x

Resolution

In this KB we will go over the steps for configuring for elasticsearch output plugin as an example:-

1) Update "output" section of fluent-bit-data-values.yaml like below example where "tls.ca_path" is defined.

    outputs: |
      [OUTPUT]
        Name              es
        Match             *
        Host             hostname
        Port              9200
        tls               on
        tls.verify        on
        tls.ca_path      /opt/certs/es_ca.crt  
        HTTP_User        username
        HTTP_Passwd      password
        Index             fluent-bit
        Suppress_Type_Name on
        Trace_Error       on
        Replace_Dots      on

2) Now we need to put the ca.cert on this particular path inside pod. For this, we firstly create secret where we will keep elasticsearch ca.crt.

kubectl create secret generic elasticsearch-ca-cert --from-file=es_ca.crt=/path/to/ca.cert/in/local/machine -n tanzu-system-logging

3) Then create below overlay secret file with name "fluent-bit-overlay-secret.yaml" to update the running fluent-bit package. This will make the change persistent inside fluent-bit pods.

apiVersion: v1
kind: Secret
metadata:
  name: fluent-bit-overlay-secret
  namespace: my-packages
stringData:
  overlays.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "DaemonSet", "metadata": {"name": "fluent-bit", "namespace": "tanzu-system-logging"}})
    ---
    spec:
      template:
        spec:
          containers:
          #@overlay/match by=overlay.subset({"name": "fluent-bit"})
          -
            #@overlay/match missing_ok=True
            volumeMounts:
            - name: tls-certs
              mountPath: /opt/certs/
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "DaemonSet", "metadata": {"name": "fluent-bit", "namespace": "tanzu-system-logging"}})
    ---
    spec:
      template:
        spec:
          #@overlay/match missing_ok=True
          volumes:
          - name: tls-certs
            secret:
              secretName: elasticsearch-ca-cert
              defaultMode: 420

 

4) Now apply the above overlay secret in namespace where fluent-bit package is installed i.e. my-packages. It will eventually help to mount the elasticsearch ca cert as volume in fluent-bit pod (controlled by Daemonset).

kubectl apply -f fluent-bit-overlay-secret.yaml

5) Eventually update the fluent-bit package using updated fluent-bit-data-values.yaml (mentioned in Step 1)

tanzu package installed update fluent-bit -p fluent-bit.tanzu.vmware.com -v 2.1.6+vmware.1-tkg.2 --values-file fluent-bit-data-values.yaml -n my-packages

6) At the end, one will able to see elasticsearch ca_cert on path /opt/certs/es_ca.crt inside fluent-bit pod.