Cluster Fails Starting cb-datagrid Service

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• One or more node's cb-datagrid service fails to startup in the cluster.

Environment

• EDR Server: All Supported Versions

Cause

Datagrid cannot communicate without Postgres

Resolution

1. Verify the cb-pgsql service is running on the primary node.

   • service cb-pgsql status

2. Confirm network communication between minion nodes and primary node, including port 5002, and 5701 are successful.
3. Ensure the cb-datagrid service isn't stopping on one of the minions.
4. If networking appears correct, this may be an issue with DNS. An option is to update DatabaseURL in /etc/cb/cb.conf to IP on all nodes and try starting services. 

Additional Information

• During cluster startup, the primary node's cb-datagrid service will wait for all other minions to start first
• Datagrid relies on the primary node's cb-pgsql  to be up and accessible across the cluster
• All nodes pull information from the cb-pgsql service on the primary node to successfully start cb-datagrid
• If step 4 corrects the issue, please reach out to your network engineer to check on the DNS servers listed in /etc/resolv.conf