Cluster status: Degraded - Druid-broker pod in CrashLoopBackOff due to Liveness/Readiness probe timeouts

Products

• VMware vDefend Firewall

Issue/Introduction

• The SSP system overview reports the cluster status as Degraded / Not stable.
• The druid-broker pod is in CrashLoopBackOff or is restarting repeatedly.
• kubectl describe pod on the broker pod shows repeated Unhealthy events for both the liveness and readiness probes.
• Analytics, dashboards, and query-backed views in SSP are slow, incomplete, or unavailable while the broker is down.

Representative events from the pod description:

Normal   Started    20m                kubelet   Started container init-conf
Warning  Unhealthy  16m (x5 over 19m)  kubelet   Liveness probe failed:
   Get "https://#.#.#.#:8282/status/health": net/http: request
   canceled while waiting for connection (Client.Timeout exceeded
   while awaiting headers)
Normal   Created    16m (x4 over 20m)  kubelet   Created container: druid
Normal   Started    16m (x4 over 20m)  kubelet   Started container druid
Warning  Unhealthy  15m (x5 over 19m)  kubelet   Liveness probe failed:
   Get "https://#.#.#.#:8282/status/health": context deadline
   exceeded
Warning  Unhealthy  14m (x10 over 19m) kubelet   Readiness probe failed:
   Get "https://#.#.#.#:8282/status/health": net/http: request
   canceled while waiting for connection (Client.Timeout exceeded
   while awaiting headers)
Warning  Unhealthy  13m (x6 over 19m)  kubelet   Readiness probe failed:
   Get "https://#.#.#.#:8282/status/health": context deadline
   exceeded
Normal   Killing    10m (x7 over 19m)  kubelet   Container druid failed
   liveness probe, will be restarted

Dignose:

• Confirm the pod state and restart count:

k get pods -A -o wide | grep -v "Run\|Com"
k get pods -A -o wide  -w | grep druid

• Review the probe failure events:

k describe pod <druid-broker-pod-name> -n nsxi-platform

• Check the druid-broker and logs for genuine errors (OOM kills, connection failures to kafka or the metadata store). If the logs show the broker starting normally and no fatal errors, the problem is probe tuning rather than a broker fault:

kubectl logs <druid-broker-pod-name> -n nsxi-platform -c druid --tail=200

• Inspect the currently configured probe values:

k get deployment druid-broker -o yaml | grep -A6 -E "livenessProbe|readinessProbe"

Environment

• SSP 5.0
• SSP 5.1.0
• SSP 5.1.1
• SSP 5.1.1.1 

 

Cause

The kubelet liveness and readiness probes for druid-broker query the broker's /status/health endpoint on port 8282. Under load — or while the broker is still starting up, building its segment/metadata view, or waiting on downstream Druid components — the endpoint does not respond within the configured timeoutSeconds window.

The two error strings in the events describe the same underlying condition: Client.Timeout exceeded while awaiting headers and context deadline exceeded both mean the probe gave up before the broker answered. Once the number of consecutive failures reaches failureThreshold, the kubelet kills and restarts the container. The restart makes the next round of probes even more likely to time out, and the pod settles into CrashLoopBackOff.

Because the broker never reports ready, SSP marks the cluster degraded. The default probe settings are simply too aggressive for this workload — the broker itself is healthy, it is just slower to answer than the probe allows.

Resolution

Please contact the broadcom support to resolve this issue. 

SSP 5.2.0 has this config change.