CrowdStrike Falcon Agent Influence on Tanzu Component Operations
Article ID: 298175
Updated On:
Products
Issue/Introduction
This Knowledge Base (KB) article aims to raise awareness on interactions observed between CrowdStrike's Falcon Agent and Tanzu Components.
CrowdStrike offers an add-on cybersecurity solution that integrates with existing systems to enhance protection against cyber threats. It leverages advanced technologies to offer real-time threat detection and response, but like any complex software integration, it may occasionally present compatibility or performance issues with the primary system it's designed to protect. This KB will review three different scenarios where CrowdStrike detected and removed files and processes in use by Tanzu components resulting in unexpected behavior.
As of this writing, there is not a partner tile on Tanzunet for CrowdStrike, however it can be implemented in a BOSH managed environment via a BOSH runtime-config. The monitored service name may show up as crowdstrike-agent or falcon-sensor.
Environment
Product Version: 4.0
Resolution
Occurrence 1
During an upgrade of TAS 4.0.9 to TAS 4.0.13, the push-usage-service errand failed with error logs:
OUT: Cell 0f353168-b883-4fc6-bda8-38ac73182981 failed to create container for instance 1472f8e1-42d0-44cb-ad42-698801d826ea: running image plugin create: making image: creating image: applying disk limits: apply disk limit: <nil>: tardis was not found in the $PATH OUT: : exit status 1
In this occurrence, CrowdStrike detected the tardis file as malware and deleted it.
Occurrence 2
Apps configured with a healthcheck type of either port or http would fail to restart or restage due to the healthcheck process being sent a SIGKILL. Once the healthcheck fails, a SIGTERM is sent to the app process. The following logs were observed:
# app logs 2024-01-22T16:29:54.61-0500 [HEALTH/0] ERR instance proxy failed to start: Failed to invoke process: ; instance proxy failed to start: Failed to invoke process: ; instance proxy failed to start: Failed to invoke process: ; Failed to invoke process: 2024-01-22T16:29:54.61-0500 [CELL/0] ERR Failed after 2.028s: startup health check never passed.
# garden logs
2024-01-22T18:25:31.858488104Z: guardian.api.garden-server.run.exited - {"handle":"5f42edc9-a728-4026-620d-c45a","id":"5f42edc9-a728-4026-620d-c45a-envoy-readiness-healthcheck-1","session":"1.1.114828","status":137}
2024-01-22T18:25:31.862135611Z: guardian.api.garden-server.run.exited - {"handle":"5f42edc9-a728-4026-620d-c45a","id":"5f42edc9-a728-4026-620d-c45a-envoy-readiness-healthcheck-0","session":"1.1.114829","status":137}
2024-01-22T18:25:31.865295143Z: guardian.api.garden-server.run.exited - {"handle":"5f42edc9-a728-4026-620d-c45a","id":"5f42edc9-a728-4026-620d-c45a-envoy-readiness-healthcheck-2","session":"1.1.114831","status":137}
2024-01-22T18:25:31.880903836Z: guardian.api.garden-server.run.exited - {"handle":"5f42edc9-a728-4026-620d-c45a","id":"dfa5f5f6-caa2-4b34-7d1b-3257deee8539","session":"1.1.114826","status":143}
2024-01-22T18:25:37.530642964Z: guardian.api.garden-server.run.exited - {"handle":"5f42edc9-a728-4026-620d-c45a","id":"5f42edc9-a728-4026-620d-c45a-envoy","session":"1.1.114825","status":137}
In this occurrence, CrowdStrike detected the healthcheck process as malicious and terminated it.
Occurrence 3
Pipelines tasks in Concourse failing when pulling images from Docker Hub due to CrowdStrike detecting “/tmp/print-metadata” in the Docker image as a vulnerability and removing the file.
Occurrence 4
KS Cluster Nodes Experience High Disk Pressure due to incompatible version of Crowdstrike Falcon causing Pods to Evict.
The root partition has a large number of core files, each one 100MB in size with the naming format core.<ID>.
Analysing the core file with gdb shows that containerd ctr command is generating the core files
gdb --core core.556774
The debuginfo package for this file is probably broken.
Core was generated by `/usr/bin/ctr -a /run/containerd/containerd.sock -n k8s.io c info 5967b9054720f7'.
Program terminated with signal SIGSYS, Bad system call.
The ctr commands are generated by Crowdstrike falcon.
Resolution is to engage Crowdstrike Falcon and upgradxe to a compatible version for the current AMI
Conclusion
When CrowdStrike is deployed within an environment and there are instances of files or processes being unexpectedly terminated or removed, it may be attributed to the security measures enforced by CrowdStrike. In such scenarios, it is advisable to examine CrowdStrike logs or consult with CrowdStrike to determine if their solution is responsible for removing these in-use items. Should it be confirmed that CrowdStrike is responsible for the deletion or termination of these files or processes, implementing a filter to exclude these items from detection might be necessary. On the other hand, if there are concerns that the affected files or processes are indeed malicious, please contact Tanzu Support.
Feedback
