EDR: How to Enable Solr Search Debug Logging

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Enable Solr debug logging to troubleshoot long running queries

Environment

• EDR Server: 7.x and Higher

Resolution

 

1. Edit /etc/cb/solr/log4j2.xml
2. Delete the comment lines shown in red, both lines above (<--) and below (-->) the debug logging line are removed

   <--
   <AsyncLogger name="com.carbonblack.cbfs.solr.handler.CbSearchRequestHandlers" level="DEBUG" /> 
   -->

3. Reproduce the search issue. If this is a watchlist timeout, the job will run every 10 minutes
4. After reproduction, use the following command to back up the debug and restore normal logging

   mv /etc/cb/solr/log4j2.xml /etc/cb/solr/log4j2.xml.debug && cp /etc/cb/solr/log4j2.xml.template /etc/cb/solr/log4j2.xml

Additional Information

• No service restart is required for Solr to pick up the new logging settings
• Since this setting is trying to find long running search queries, logs for troubleshooting should be collected no sooner than 30 minutes after reproducing the issue. 
• Debug logging will list all queries active and completed every 10 seconds
• Restarting the services will also cause EDR to replace the existing log4j2.xml with the log4j2.xml.template
• For Solr debugging to survive a restart of the services the log4j2.xml.template can be modified