Audit and Remediation: Memory and Performance Issues

Products

• Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

• High memory usage after query start
• Domain Controllers harder hit when user-based LiveOps queries are run
• Long-running queries will consume system resources until host and/or osquery process crash.

Environment

• Carbon Black Cloud Sensor: 3.5.0.1627 and Lower
  • Audit and Remediation 
• Windows Server: All supported versions 

Cause

• Osquery bug
• Sensor logic not optimal

Resolution

Upgrade to sensor 3.5.0.1680 or Higher
 

Additional Information

• Server environments with a large number of local users are harder hit by this
• Running Queries of this nature will result in excessive memory usage and should be avoided:

  SELECT * FROM users JOIN chrome_extensions USING (uid);

• OS Query Bug: https://github.com/osquery/osquery/issues/6411
• DSEN-8537 added a timeout and memory cap on queries
• Despite the users table spec saying that uid is an index, the Windows implementation of that table doesn't support that column to be an index and so it doesn't optimize the query based on it.
• When a column is an index and a JOIN is done on it, what is expected is that the table will do an optimization and only search and return that column value.  For each row to JOIN, sqlite will call the table row generation, so it has to be fast. Instead here every time the table generation code is called, all the users are retrieved and almost all then discarded later by sqlite, because only one uid is really needed.