How to Find Who or What is Causing Writer Core Lock

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

How to setup audit logging to investigate a cause of the write core lock file being touched, causing corruption. 

Environment

• Carbon Black EDR Server: All supported versions

Resolution

1. Start up audit logging: 

   systemctl start auditd

2. Use auditctl to create a watch for any write, append or execute:
   Note: cbevents_core needs to be the actual cbevents_<data> core. Auditctl does not accept wildcards, so make sure to create a new watch for every new writer core that get's created 

   auditctl -w /var/cb/data/solr/cbevents/<cbevents_core>/data/write.lock -p war -k password-file

3. When the issue is seen again, run the following command to capture the output: 

   ausearch -k solrlock

Additional Information

The following error in Solr can be seen when Writer core lock happens:

Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at xxxx-xx-xxTxx:xx:xxxxZ, (lock=NativeFSLock(path=/var/cb/data/solr/cbevents/cbevents_xxxx_xx_xx_xxxx/data/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:xxxxxxx exclusive valid],creationTime=xxxx-xx-xxTxx:xx:xxxxZ))