EDR: Error Message When Starting Services: 'fatal error wont start.'

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• One or more EDR services will report 'fatal error wont start' when running 'service cb-enterprise start' command.
• All other services are able to be manually started.
• Messages similar to the following are reported in the /var/log/messages file:

SELinux is preventing rhsmcertd-worke from read access on the file CarbonBlack.repo. For complete SELinux messages run: sealert -l 4255cebf-007c-453e-bce7-6df300811b45
SELinux is preventing rhsmcertd-worke from read access on the file carbonblack-alliance-client.crt. For complete SELinux messages run: sealert -l 4255cebf-007c-453e-bce7-6df300811b45

kernel: EXT4-fs error (device dm-2): ext4_validate_block_bitmap:376: comm rsyslogd: bg 13742: bad block bitmap checksum
kernel: EXT4-fs error (device dm-2): ext4_validate_block_bitmap:376: comm BESClient: bg 9317: bad block bitmap checksum


kernel: EXT4-fs (dm-2): Delayed block allocation failed for inode 104859516 at logical offset 678 with max blocks 2 with error 117
kernel: EXT4-fs (dm-2): This should not happen!! Data will be lost

Environment

• EDR Server: All Versions

Cause

Messages reported are due to a corrupt file system.

Resolution

Restore from a known good backup, or work with OS admin/vendor to resolve corrupt file system. 

Additional Information

• If cb-pgsql service does not start, messages similar to the following are reported in /var/log/cb/pgsql/postgresql.log: 
   

  LOG: database system shutdown was interrupted
  LOG: 127.0.0.1(49038)] FATAL: the database system is starting up
  LOG: database system was not properly shut down; automatic recovery in progress
  LOG: redo starts at 6/F79E4688
  FATAL: the database system is starting up
  LOG: invalid record length at 6/F8B62FF0: wanted 24, got 0
  LOG: redo done at 6/F8B62FC8
  LOG: last completed transaction was at log time 2020-06-06 07:01:21.113299-07
  LOG: checkpoint starting: end-of-recovery immediate
  PANIC: could not flush dirty data: Structure needs cleaning