DLP Detection servers show "Unknown" status

search cancel

DLP Detection servers show "Unknown" status

book

Article ID: 160481

calendar_today

Updated On:

Products

Data Loss Prevention Enforce

Issue/Introduction

This is a master troubleshooting article that covers various situations in which one or more Symantec Data Loss Prevention (DLP) Detection servers display an "Unknown" status. For example:

In the DLP Enforce Server console, under System > Servers and Detectors > Overview, one or more Detection servers have a status of "Unknown".
After adding additional Detection server(s) into your existing DLP environment, all detection servers change to 'Unknown' status after a MonitorController service restart.
After Enforce and Detection server upgrades are complete and all services are running, the status of the Detection servers will switch back and forth between a "running" and "unknown" state.
- To test this, navigate to System > Servers and Detectors > Overview page and click on the Enforce console refresh button a few times.
- You should see the Detection server status change with each refresh.
You log into your Enforce console, click on a detection server with "Unknown" status, and filereader shows a status of "Running Selected" or "Down"
After upgrading Enforce, before upgrading the detection servers, the status of the Detection servers may show 'Unknown'

NOTE: For Cloud Detectors showing as "disconnected" or in "unknown" status, please see this article: Landing page: DLP cloud detectors disconnected (broadcom.com)

Depending on the situation, you may see the following errors:

Network Related Issues:

Enforce server Monitor Controller log:

FINE: failed to finish connection to: test:incidentwriter:[IP ADDRESS]:8100

com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information ....... Caused by: java.net.ConnectException: Connection timed out: no further information ...... FINE: connect operation in SSLConnect failed com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information ........ May 12, 2010 11:57:06 AM com.vontu.communication.transport.ConnectWrapperOperation select INFO: connectOp select failed

Detection Server:

VontuMonitor log:

INFO | jvm 12 | 2012/03/22 02:47:04 | WrapperSimpleApp: Encountered an error running main: INFO | jvm 12 | 2012/03/22 02:47:04 | WrapperSimpleApp: com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.

BoxMonitor log:

16/03/2012 2:25:17 AM com.vontu.boxmonitor.BoxMonitor main| SEVERE: (BOXMONITOR.2) BoxMonitor failed to startup com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.

You may also see:
Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager <init> SEVERE: failed to init server channel channel manager Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager SEVERE: failed to init server channel channel manager java.net.BindException: Cannot assign requested address

SSL/Keystore Issues:

On the Detection server the boxmonitor log will show the following:

INFO: Operation com.vontu.communication.transport.AcceptWrapperOperation:1481653881505:null:com.vontu.communication.transport.SessionIdentifier@708fc9e5 failed with exception: com.vontu.communication.transport.exception.TransportException: javax.net.ssl.SSLHandshakeException: no cipher suites in common com.vontu.communication.transport.ChannelManager handleOperationFailure INFO: removing session from cache: com.vontu.communication.transport.SessionIdentifier@708fc9e5 for id: null

After Adding additional Detection Servers:

The monitor controller log shows:

XXX XX, XXXX XX:XX:XX XX com.vontu.communication.transport.OperationManager enqueue INFO: blocking on op mgr queue

Resolution increase java heap maxmemory to support the increased detection servers. Java heap maxmemory should be 4 GB per 12 detection servers.

Filereader status of "Running Selected" or "Down":

FileReader log on the Detection server shows an error: "no disk space" and no additional entries are written to the log.

All detection servers show as unknown status in an DLP environment primarily aimed at Discover scanning.

Cause

Network Connectivity Issues
SSL/Keystore Issues
Resource or Tuning Issues
Large number of Detection Servers
- In environments with a large number of Detection servers, it is possible to 'overload' the Enforce queue which controls operations across all transport connections. In instances where this occurs, an 'Unknown' status can be seen across all Detection servers, often proceeding the addition of a new Detection server into the environment.
Detection servers switching between 'running' and 'unknown' state:
- The upgrade process on the Enforce server never completed and is making connections to the Detection servers on the same port used by monitor controller to receive status updates from the Detection servers.
Filereader status of "Running Selected" or "Unknown"
- Detection server disk space is at 100% used.
Insufficient maximum java heap memory assigned to the SymantecDetectionServerController process.
Enforce server was upgraded to newer version and the services on the Detection Servers were restarted before they were upgraded to the same version

Resolution

Network Connectivity Issues:
- Use the nslookup command as follows (same for both Windows and Linux):
- Communication port test
SSL/Keystore Issues:
Resource or Tuning Issues:
Large Number Of Detection Servers:
Detection servers switching between 'running' and 'unknown' state:
Filereader status of "Running Selected" or "Unknown"
After upgrading Enforce to a newer version, Status of Detection Servers is "Unknown"
Insufficient memory allocated to SymantecDetectionServerController process.
Additional information

Network Connectivity Issues:

Follow the steps below to diagnose host unreachable, name resolution and IP filtering issues:

Confirm that the host or fully qualified domain name (FQDN) of the detection server used in the Enforce UI configuration settings
Start a command window (windows) or terminal window (Linux) on the Enforce server.
Ping the IP address of the detection server; use the correct IP for 10.10.10.1
- ping 10.10.10.1

Possible Results	Next Steps
Error message that ping "...could not find host..."	There is a name resolution problem. Consult with your Network Administrator to resolve the issue.
100% packet loss (all ping requests timed out)	Detection server is not reachable. Correct network and routing issues and repeat the ping test.
Packet loss (some ping requests timed out)	Network issues between the Enforce server and detection server. Correct network and routing issues and repeat the ping test.
0% packet loss	Continue to the next section.

Use the nslookup command as follows (same for both Windows and Linux):

Example commands

nslookup mon1

nslookup mon1.some.corp.com

Example Output

Server:
Address:
Name: mon1
Address: 10.10.10.1

Server:
Address:
*** mon1 can't find mon1: Non-existent domain

Possible Results	Next Steps
Discrepancy between Name and IP in DNS record, with Enforce Detection Server settings.	Should there be a difference in the query result and the server name or IP, contact your DNS Administrator to resolve the issue.

A short term solution is to add the host name or FQDN to the Enforce server local host file.

Windows: c:\windows\system32\drivers\etc\hosts

Linux: /etc/hosts

Example content:

10.10.10.1 mon1 mon1.some.corp.com

Communication port test

From the Enforce server, use a telnet client to connect to the detection servers default communication port, 8100.

telnet mon1 8100

Possible Results	Next Steps
...Could not open connection...	Ensure port 8100, inbound from Enforce to Detection Server is allowed through any Network or Host based firewalls.
...Connection refused...	Port 8100 may be opened under a different application. Consult with your Systems Administration team to determine if there is a port conflict with another application.
Telnet session Could not open connection to the host, on port 8100: Connect failed	At the detection server, ensure the Symantec Detection Server service is running. Confirm the port 8100 is open and status is reported as LISTENING using the `netstat` command. Note: In the case of network monitor there will be multiple IP addresses, ensure that the IP listening on port 8100 is not the capture card IP address. If it is then update the communication.properties file in the config folder and set the bind address.

If the above steps do not resolve the issue, try the following:

On the detection server, open \Protect\Config\Communication.properties in a text editor.
- By default serverBindName = is commented out. e.g.:
- # Transport bind address.
  # The IP address of the NIC that transport server sockets and the registry should be bound to;
  # The sockets will be bound to the default IP address on the machine if this value is not specified.
  # serverBindName =
If this is the case, then the Monitor service will bind to the default IP address.
If you see the old IP address entered in the "serverBindName=" setting you can either comment it out or enter the new IP address if you want the Monitor service to bind to that specific IP Address.
If needed, Update the "serverBindName=" setting.
Restart the Vontu Monitor service on the detection server:
1. Start > Run > services.msc
2. Right click the Vontu Monitor services and select Restart.
Now check the Enforce console and you should see the detection server is communicating.

Additional Solutions:

Verify there are no Network Access Control Lists (ACLs) restricting permissions to the account used by the DLP services.
1. On the detection server, Start > Run > services.msc
2. Right click each of the DLP services and choose properties.
3. Select the Log On tab.
4. Note the account being used by the service.
5. Verify there are no permission restrictions configured in your environment that are applied to this account.
If the Detection is running on Linux server, verify the IPTABLES and make sure the port 8100 is allowed.
For additional assistance with this step please contact your vendor.

------------------------------------------------------------------------------------------

SSL/Keystore Issues:

Each server should have its own keystore generated when you first add the server to Enforce.

You can verify this by checking the keystore directory on Enforce \SymantecDLP\Protect\keystore for a file that matches this format:

enforce..sslKeyStore
monitor####.timestamp.sslKeyStore

The number at the end of the monitor name will most often reflect the monitor ID of the detection server this belongs to.

To identify the monitor ID, mouse-over the detection server in server overview and note the javascript information noted at the bottom of the screen.

The name of the keystore file is case sensitive, so ensure when copying from one OS to another that this is preserved - e.g.,"monitor.timestamp.sslKeyStore".

If not, a Linux Monitor may report the following failure when starting the VontuMonitor service:

Log message	Log	Resolution
"Unable to resolve the full path of the configuration file, start: No such file or directory"	VontuMonitor.log - Detection server	Ensure the protect user has permissions to the /keystore directory on both Enforce and the effected detection server. Ensure the proper keystore file exists in the /keystore directory.
javax.net.ssl.SSLHandshakeException: no cipher suites in common	VontuMonitor.log - Detection server	Check the SSLcipherSuite listing in MonitorController.properties on Enforce and Communication.properties on the detection server have SSL algorithms in common
failed with exception: com.vontu.communication.transport.exception.TransportException: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) com.vontu.communication.transport.ChannelManager handleOperationFailure	boxmonitor log on Network Monitor	Check the SSLcipherSuite listing in MonitorController.properties on Enforce and Communication.properties. Modifying the “SSLautonegotiate” from “false” to “true”

Resource or Tuning Issues:

Monitor Running out of memory

Within the VontuMonitor log, similar errors to the following error can be seen:

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 59136 bytes for Chunk::new

JVM exited unexpectedly.

This is indicative that SymantecDLPDetectionServerService does not have enough memory to function properly.

On the Detection server experiencing the issue, navigate to \Program Files\Symantec\DataLossPrevention\DetectionServer\Services (Windows) or /opt/Symantec/DataLossPrevention/DetectionServer/Services (Linux) and open SymantecDLPDetectionServerController.conf
Take note of the following Java Heap Size values:

wrapper.java.initmemory
wrapper.java.maxmemory

By default, these values are set to 128/256 respectively. When increasing these values, follow these best practice guidelines:

Always increase memory values as a factor of 128 (i.e: 256, 512, 1024).
Always keep the maxmemory value twice the value of initmemory, and neither value should ever be greater than the available RAM on the server.

Restart the SymantecDLPDetectionServerService service.
If the issue continues, restart the Detection server. This may release any resources being held by the operating system.

Symantec DLP Detection Server Controller service resources are limited

In environments with a large number of Detection servers, it is possible to 'overload' the Enforce queue which controls operations across all transport connections.

In instances where this occurs, an 'Unknown' status can be seen across some or all Detection servers, often proceeding the addition of a new Detection server into the environment.

The following steps with increase the transport operation queue size.

NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.

On the Enforce server, navigate to \Program Files\Symantec\DataLossPrevention\EnforceServer\<DLP_version>\Protect\config (Windows) or /opt/Symantec/DataLossPrevention/DetectionServer/<DLP_version>/Protect/config (Linux).
Open the file, MonitorController.properties, in a text editor.
Locate the configuration key, maxQueueSize.
- This setting is 5000 by default, and controls the maximum number of operations that can be pushed out across all transport connections.
Change the value of maxQueueSize to 10000.

A value this large is enough to handle the number of operation requests seen in large environments, without having a performance impact on memory.

Restart the SymantecDLPDetectionServerControllerService service.

Monitor Controller configured memory limits are too low

The following error is seen in VontuMonitorController.log(Linux Only):

java.lang.OutOfMemoryError: unable to create new native thread

If the out of memory error includes, "unable to create new native thread," this is more likely due to a limit of how many processes a single user can run at one time on RedHad linux.

Log into the Enforce system as root level user.
Switch to root; su - root
Run the following command to print the limits file to the screen: cat /etc/security/limits.conf
Review the limits.d file for any explicit limits to the protect user.

If the protect user is not restricted explicitly, add the following to the limits.conf file to raise the process limit above the default of 1024.

protect soft nproc 4096 protect hard nproc 63536

Service Crashing

Within the VontuMonitor.log on the Detection Server, the following error repeats:

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate  bytes for ChunkPool::allocate

# An error report file with more information is saved as:

# \Protect\bin\hs_err_pidNNNN.log

JVM exited unexpectedly.

3rd-party applications can prevent Java.exe from starting under normal conditions. It is recommended you collect the hs_err_pidXXX files and contact support.

Large Number Of Detection Servers:

The following steps will increase the transport operation queue size.

NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.

On the Enforce server, navigate to \Program Files\Symantec\DataLossPrevention\EnforceServer\<DLP_version>\Protect\config (Windows) or /opt/Symantec/DataLossPrevention/DetectionServer/<DLP_version>/Protect/config (Linux).
Open the file, MonitorController.properties, in a text editor.
Locate the configuration key, maxQueueSize.
This setting is 5000 by default, and controls the maximum number of operations that can be pushed out across all transport connections.
Change the value of maxQueueSize to 30000.
A value this large is enough to handle the number of operation requests seen in large environments, without having a detrimental impact on memory.
Save and close the file.
Restart the MonitorController service.
All Detection Servers should report in to the Enforce console within a short time period.

Detection servers switching between 'running' and 'unknown' state:

Stop the upgrader process:

Open task manager on the Enforce server:
1. Start > Run > taskmgr
Find the process named "upgrader-java"
Right-click the "upgrader-java" process and choose "end process".

Filereader status of "Running Selected" or "Unknown"

Check if the disk space on the Detection Server is full:
1. Log in locally on the Detection server
  - Start > Run > diskmgmt.msc (Windows)
  - df -h or df -a (Linux)
If disk space is at 100%, free up space on the drive.
Restart DLP services on the Detection server.

After upgrading Enforce to a newer version, Status of Detection Servers is "Unknown"

1. If the services on the Detection Server have been restarted before upgrading to the same version, the only solution is to upgrade the Detection Servers to the same version of Enforce to get them to reconnect.

Insufficient memory allocated to SymantecDetectionServerController process.

It can be observed in DLP environments where the customer is a heavy discover scanning user that runs a large number of incremental scans on a daily/weekly basis and also runs a high number of discover scans concurrently a significant amount of java heap memory needs to be allocated to SymantecDLPDetectionServerController process, not doing so may result in the SymantecDLPDetectionServerController process on Enforce stopping and all detection servers showing as an unknown status.

Restarting the service does not resolve the issue and the service is likely to stop again after some time. This may be a silent failure with little or no useful information found in Enforce logs that will pinpoint the issue, the best way to observe the issue is to monitor the memory consumption of the respective DLP processes on the Enforce server and note which ones are running near or at max allocation.

Note: Running a java process near max allocation will perform poorly and consume excessive CPU due to garbage collection and should be avoided. If the process is run near or at max heap allocation, the max allocation should be increased giving consideration to available RAM on the server and the heap allocation of other DLP services/processes.

Feedback

thumb_up Yes

thumb_down No