EDR: OS and CB processes very slow. Error starting Response services "Waiting for cb-rabbitmq to initialize...Key v1 not found in hazelcast"

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Error starting CB Response services

Starting cb-redis: [ OK ]
Starting cb-rabbitmq: [ OK ]
Waiting for cb-rabbitmq to initialize...Key v1 not found in hazelcast
Apr 13, 2020 07:54:33 AM HazelcastClient.HeartbeatService

• Takes 15 minutes to start or stop CB Response services manually.
• /usr/share/cb/cbdiag takes a very long time to complete
• ssh or putty into the server takes 30 seconds to login
• /var/log/cb/nginx access log in nginx may show 499 and 503 errors in bursts

 

Environment

• EDR Server: All versions
• EDR Sensor: All versions
• RHEL: All versions
• CentOS: All versions

Cause

/var/log/messages shows a Firmware Bug is occurring:
 

Jun 1 14:38:56 jh-dsw-cbhead-1 kernel: [Firmware Bug]: No valid trip found
Jun 1 14:38:56 jh-dsw-cbhead-1 kernel: ERST: Error Record Serialization Table (ERST) support is initialized.

 

Resolution

Contact the operating system support and request a firmware patch to address the ERST error.