EKS cluster nodes experiencing high disk pressure due to Crowdstrike falcon

Products

• VMware Tanzu Platform - TAP
• VMware Tanzu Mission Control Self-Managed

Issue/Introduction

KS cluster nodes experience high disk pressure and several pods are in Evicted state.

This impacts all Tanzu related workloads on the cluster including and TAP and TMC.

Environment

AWS EKS cluster with Crowdstrike Falcon

Cause

The root partition has a large number of core files, each one 100MB in size with the naming format core.<ID>.
Analysing the core file with gdb shows that containerd ctr command is generating the core files

gdb --core core.556774
The debuginfo package for this file is probably broken.
Core was generated by `/usr/bin/ctr -a /run/containerd/containerd.sock -n k8s.io c info 5967b9054720f7'.
Program terminated with signal SIGSYS, Bad system call.

 

The ctr commands are generated by Crowdstrike falcon.
However the version of falcon is incompatible with the EKS AMI version

Resolution

Engage Crowdstrike Falcon and AWS EKS support and upgrade to a compatible falcon version for the current EKS AMI