EDR: SOLR Writers Hanging Due To "Too Many Open Files" Error

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Solr failing with a "Too many open files" error

  • Too many close [count:-8] on org.apache.solr.core.SolrCore@7057d1d. Please report this exception to [email protected]

    
     
• Solr cores may not roll over when expected

Environment

• EDR Server: 6.2.3 - 7.1.x (formerly CB Response)
• RHEL: 7.x
• CentOS: 7.x

Cause

Setting limit for max number of open files does not work (CB-27866). The supervisord takes control and supervisord.conf setting of mindfs takes precedence.

Resolution

• Upgrade to sever version 7.2 or higher
• The following workaround can be used in the interim: 

1. Create backup of /etc/cb/supervisord.templates/cb-supervisord.conf.template
2. Edit /etc/cb/supervisord.templates/cb-supervisord.conf.template
3. Change minfds from 4096 to 80000

   • loglevel=info ; (log level;default info; others: debug,warn,trace)
     pidfile=/var/run/cb/cb-supervisord.pid ; (supervisord pidfile;default supervisord.pid)
     nodaemon=false ; (start in foreground if true;default false)
     minfds=80000 ; (min. avail startup file descriptors;default 1024) 
     minprocs=200 ; (min. avail process descriptors;default 200)

4. Restart cb-enterprise services