ICDx collector stops.

Products

• Integrated Cyber Defense Exchange

Issue/Introduction

The collector stops and the user has to start it again manually.

Environment

ICDx SICDM EDR/SICDM Collector behavior:

The collector processes the events based on the HTTP response it received from the upstream API server. 

• For some response, it does retry e.g 502,500
• For some response, it terminates/shut down the process e.g 401,429 etc

Cause

Most of the time collector stops if it will make to many restarts attempt in a specific time. Logs would point to similar entries:

[ProcessQueue1] ERROR c.s.dx.launcher.ApplicationSupervisor - Process 'NLSESC' has been stopped, too many restarts

Examine the log further if you see multiple logs similar to those at the same time:

WARN  com.symantec.http.support.HttpRequester - poll request attempt 1 - potentially recoverable failed request (will retry in 982ms): 502 - Bad Gateway

In such case most probaly proxy is being used between ICDx and API server. 
The HyperText Transfer Protocol (HTTP) 502 Bad Gateway server error response code indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server.

Resolution

Examine the proxy server for possible reasons of 502 responses.

Additional Information

Logs to be check while troubleshooting are sicdm_col_dx-NLSESC and launcher_dx