Cb Response: Cluster / node fails to start with "[Errno 30] Read-only file system: '/var/cb/data/solr5/./solr.xml'

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Standard output from "/usr/share/cb/cbcluster start" shows errors for a node:

Node[ 3] ==== machine_3 ==== 
Node[ 1 ] ==== machine_1 ==== 
Node[ 2 ] ==== machine_2 ==== 
Node[ 3] [Errno 30] Read-only file system: '/var/cb/data/solr5/./solr.xml' 
Node[ 3] Traceback (most recent call last): 
Node[ 3] File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/exceptions.py", line 70, in decorator 
Node[ 3] File "/usr/share/cb/virtualenv/lib/python2.7/site-packages/cb/maintenance/cbstartup/main.py", line 165, in main 
Node[ 3] File "/usr/share/cb/virtualenv/lib/python2.7/site-packages/cb/maintenance/cbstartup/main.py", line 111, in run 
Node[ 3] File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/solr_core_setup.py", line 7, in execute 
Node[ 3] File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/solr_core_setup.py", line 18, in execute 
Node[ 3] File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/solr_core_setup.py", line 46, in _deploy_files_to_data_dir 
Node[ 3] File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/co 
Node[ 3] reservices/src/cb/utils/solr_core_setup.py", line 78, in _deploy_files 
Node[ 3] File "/usr/lib64/python2.7/shutil.py", line 133, in copy 
Node[ 3] copyfile(src, dst) 
Node[ 3] File "/usr/lib64/python2.7/shutil.py", line 97, in copyfile 
Node[ 3] with open(dst, 'wb') as fdst: 
Node[ 3] IOError: [Errno 30] Read-only file system: '/var/cb/data/solr5/./solr.xml'

Environment

• Cb Response Server: 5.x.x and Higher
• Linux: All Supported Versions

Cause

• The underlying Linux file system has an integrity issue, and is likely failing file system consistency check, and Linux mounts the partition as "read-only"

Resolution

1. Check with the Linux administrator on how to check and repair file systems with "fsck" utility.

Additional Information

• The syntax for the fsck utility can be found here: https://docs.oracle.com/cd/E19455-01/805-7228/6j6q7uf0j/index.html 

  • Example: sudo fsck -nf /dev/<mount>