Sensor Uploading Service is down
Article ID: 409797
Updated On:
Products
Issue/Introduction
The Sensor Uploading Service is in down state. This service is responsible for processing and sending all critical security events (like IDS alerts, malware analysis results, and network flows) to the Security Services Platform (SSP). A down state means events and flows are not being reported to SSP.
Environment
vDefend SSP >= 5.1
NDR Sensor >= 5.1
Cause
The Sensor Uploading Service being down means its core processes have failed to start or have terminated. The primary causes for this are:
1. Service failed to start: The service may have failed during its startup sequence.
2. Dependency failure: The service has critical dependencies on rabbitmq, sensor-container-orchestration and docker services. If either of these essential services is not running, the sensor-uploading service will fail to start.
Resolution
Sensor uploading service not starting up could be temporary and has been designed to restore on its own.
If this does not happen after 30 mins, then it might be worth checking the below troubleshooting steps.
Please follow these troubleshooting steps from the NDR Sensor CLI using admin credentials to diagnose and resolve the issue
1. Confirm the service is Down: Access the NDR Sensor CLI and check the status of the service. The service will likely be in "stopped" state.
ndr-sensor> get service sensor-uploading
2. Attempt a manual restart: The first step is to try and bring the service up manually.
ndr-sensor> restart service sensor-uploading
Wait for 15-20 mins and then check the status again. If it fails to start and remains stopped, proceed to the next steps.
3. Verify critical dependant services: Ensure the core dependencies are operational. The sensor-uploading service will not start if these are down.
(a) Docker is used to run the core application services
ndr-sensor> get service docker
If docker service is down then, try restarting the appliance.
(b) sensor-container-orchestration is the central logic that manages the lifecycle of all the other containerized application services, making sure the sensor is always running the right components based on its current configuration and status
ndr-sensor> get service sensor-container-orchestration
If sensor-container-orchestration is down, then try restarting the service
ndr-sensor> restart service sensor-container-orchestration
(c) rabbitmq acts as the central message broker for the entire system
ndr-sensor> get service rabbitmq
Attempt to restart the rabbitmq service, if its not running
ndr-sensor> restart service rabbitmq
(d) sensor-health detects the health of each of the core services.
ndr-sensor> get service sensor-health
Attempt to restart the sensor-health service if it is down.
ndr-sensor> restart service sensor-health
and then try restarting the sensor-uploading service again.
If the issue still persists, then it is advisable to collect the NDR Sensor support bundle (refer documentation for how to collect support bundle) and raise a support ticket
Feedback
