EDR: Out of Memory Error when running '/usr/share/cb/cbcheck sensor-builds --update'

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Version of the error below is encountered when attempting to import a new sensor version into the EDR instance:
  • When attempting to import the new package:

sudo /usr/share/cb/cbcheck sensor-builds --update
(psycopg2.errors.OutOfMemory) out of memory
DETAIL: Failed on request of size 67108864.

[SQL: INSERT INTO sensor_build_packages (name, data, sha256) VALUES (%(name)s, %(data)s, %(sha256)s)]
[parameters: {'name': 'CarbonBlackClientSetup-linux-v7.0.1.14543.rpm', 'data': <psycopg2.extensions.Binary object at 0x7f597441ba50>, 'sha256': '12a62b3f8baa17c16b633e93bdc43efa9179fdcd19159738aa4f8ab97e2703db'}]
(Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
self.dialect.do_execute(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.OutOfMemory: out of memory
DETAIL: Failed on request of size 67108864.

• /var/log/cb/pgsql/postgresql-<date>.log:

Traceback (most recent call last):
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/exceptions.py", line 84, in decorator
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/cli_cmd.py", line 165, in decorator
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbcheck/main.py", line 47, in main
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/cli_cmd.py", line 35, in run_command
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbcheck/commands/sensor_builds/cmd_main.py", line 48, in execute
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/maintenance/cbcheck/commands/sensor_builds/cmd_main.py", line 77, in _update
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/sensor_builds.py", line 23, in update_db
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/sensor_builds.py", line 58, in _update_sensor_builds_db_to_match_fs
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/sensor_builds.py", line 140, in _add_build_version_by_filename
File "/usr/share/cb/virtualenv/lib/python3.8/site-packages/cb/utils/sensor_builds.py", line 209, in _create_build_package_record
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 1036, in commit
self.transaction.commit()
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 503, in commit
self._prepare_impl()
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl
self.session.flush()
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2496, in flush
self._flush(objects)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2637, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 2597, in _flush
flush_context.execute()
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
_emit_insert_statements(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1083, in _emit_insert_statements
c = cached_connections[connection].execute(statement, multiparams)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_clauseelement
ret = self._execute_context(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1287, in _execute_context
self._handle_dbapi_exception(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
util.raise_(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
self.dialect.do_execute(
File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.OutOfMemory) out of memory
DETAIL: Failed on request of size 67108864.

[SQL: INSERT INTO sensor_build_packages (name, data, sha256) VALUES (%(name)s, %(data)s, %(sha256)s)]
[parameters: {'name': 'CarbonBlackClientSetup-linux-v7.0.1.14543.rpm', 'data': <psycopg2.extensions.Binary object at 0x7f597441ba50>, 'sha256': '12a62b3f8baa17c16b633e93bdc43efa9179fdcd19159738aa4f8ab97e2703db'}]
(Background on this error at: http://sqlalche.me/e/e3q8)

Environment

• EDR Server: All Supported Versions

Cause

• Server likely doesn't meet OER recommendations or is out of memory when attempting to import sensor package into PGSQL.

Resolution

• Add additional memory based on OER Recommendations