EDR: Receiving 502 and 504 responses when accessing console after upgrade to 7.5.1

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Attempts to sign in return 502 and 504s
• Nginx logs on minions show that event data is ingesting, but Node 0 shows mostly 502 and 504 responses
• Excessive API queries from third party scripts may be sent to the server
• Datagrid shows "Connection reset by peer" errors
• Coreservices shows heartbeat issues with datagrid

  • ss=('localhost', 5701), id=0) has stopped heart beating.", None)

Environment

• EDR Server: 7.5.1
• Large Cluster

Cause

Something is wrong with DNS which causes the delays in requests with the default resolver - c-ares

Resolution

Disable c-ares

• Edit /etc/cb/cb.conf
• Add the line
  • UseGeventAresResolver=false
• Restart services