EDR Server: Start hangs at Coreservices

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Bad file descriptor message in /var/log/cb/coreservices/coreservices.log

  • File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/gevent/_socket3.py", line 66, in _dummy
        raise OSError(EBADF, 'Bad file descriptor')
    OSError: [Errno 9] Bad file descriptor

• Other errors in /var/log/cb/coreservices/coreservices.log stack point to FileNotFoundError

  • Traceback (most recent call last):
      File "/usr/lib64/python3.8/logging/handlers.py", line 855, in _connect_unixsocket
        self.socket.connect(address)
      File "/usr/share/cb/virtualenv/lib64/python3.8/site-packages/gevent/_socket3.py", line 421, in connect
        raise error(result, strerror(result))
    FileNotFoundError: [Errno 2] No such file or directory

• Coreservices service repeatedly stopping and starting in process list with different PIDs

Environment

• EDR Server: All versions
• RHEL 6.10

Cause

One of the name servers in resolv.conf is not accessible. 

Resolution

1. Edit /etc/resolv.conf
2. Remove any name servers not available or having performance issues
3. Restart services

Additional Information

• nslookup of name servers can be a quick check to confirm reachability, but if the name server has performance issues, it can still cause startup failures. 
• One of the python packages used by EDR tests that name servers can be resolved during startup. If one fails, start up can fail in various ways on different services.