vDefend SSP Alarm: Security Services Platform cluster disk usage is high or very high

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

You are running SSP 5.0 or later and have encountered an alarm with the description:
"The disk usage of the Kubernetes cluster is currently {{ .Value }}%, which exceeds the threshold value."

This indicates that one or more nodes in the Kubernetes cluster are experiencing high disk usage, which can impact cluster operations, such as scheduling new pods, storing logs, and managing workloads.

Environment

vDefend SSP Version: 5.0 and later

Cause

High disk usage on worker nodes can occur due to various reasons:

1. Insufficient disk space: The allocated disk space may be inadequate for the current workload or data growth.
2. Disk-intensive operations on that node

Resolution

1. On the SSPI UI, Navigate to Lifecycle Management → Instance management . Select Edit Deployment Size and increment the worker node count by +1.
2. Once scale is done, Validate that new nodes are operational.
   

   k get nodes

   Expected output : 
NAME           STATUS   ROLES    AGE     VERSION
node-1         Ready    <role>   xxm     v1.xx.x
node-2         Ready    <role>   xxm     v1.xx.x
new-node       Ready    <role>   xxm     v1.xx.x  # Ensure the new node is Ready 

3. Adding new node should increase the overall disk size used to compute this alarm.
4. If issue persists open a ticket with Broadcom.