Steps to remove the Elastic Search Indexes when Kubernetes PVC ( Persistent Volume Claim) is full in VIP AuthHub

Products

• VIP Authentication Hub

Issue/Introduction

This KB article provide steps to clear the Elasticsearch indexes when persistent volume claim becomes full so no space is left on the device.

Environment

VIP Authentication/Auth Hub

Resolution

Following steps can be done to clear the Elasticsearch indexes to regain the space:

• Scaling down the Fluent-Bit Daemonset to zero. A Daemonset cannot be scaled down So, a node selector is used that does not match an existing node in order to stop the daemon set instances

                   $ kubectl patch \
         "$(kubectl get daemonsets.apps --namespace "${SSP_NAMESPACE}" --selector 'app.kubernetes.io/name=fluent-bit' --output name)" \
         -n "${SSP_NAMESPACE}" \
         -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'

• Remove the Elasticsearch  indexes following the below command. Please note that ES_PWD needs to be set to the Elasticsearch elastic user's password below.

     $  kubectl exec -n "${ES_NAMESPACE}" "elasticsearch-es-default-0" -c "elasticsearch" -- curl --insecure -u"elastic:$ES_PWD" -s -XDELETE "https://localhost:9200/ssp_audit
   $ kubectl exec -n "${ES_NAMESPACE}" "elasticsearch-es-default-0" -c "elasticsearch" -- curl --insecure -u"elastic:$ES_PWD" -s -XDELETE "https://localhost:9200/ssp_log
   $ kubectl exec -n "${ES_NAMESPACE}" "elasticsearch-es-default-0" -c "elasticsearch" -- curl --insecure -u"elastic:$ES_PWD" -s -XDELETE "https://localhost:9200/ssp_tp_log

• Scale back the Fluent-Bit Daemon set

      $ kubectl patch \
         "$(kubectl get daemonsets.apps --namespace "${SSP_NAMESPACE}" --selector 'app.kubernetes.io/name=fluent-bit' --output name)" \
         -n "${SSP_NAMESPACE}" \

   -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'

  Note:->  Please make sure that the ES_NAMESPACE ( Elasticsearch Namespace) and SSP_NAMESPACE environment variables are set before running the above commands.