DLP Detection servers show "Unknown" status

book

Article ID: 160481

calendar_today

Updated On:

Products

Data Loss Prevention Enforce

Issue/Introduction

This is a master troubleshooting article that covers various situations in which one or more Symantec Data Loss Prevention (DLP) Detection servers display an "Unknown" status. For example:

  • In the DLP Enforce Server console, under System > Servers and Detectors > Overview, one or more Detection servers have a status of  "Unknown".
  • After adding additional Detection server(s) into your existing DLP environment, all detection servers change to 'Unknown' status after a MonitorController service restart.
  • After Enforce and Detection server upgrades are complete and all services are running, the status of the Detection servers will switch back and forth between a "running" and "unknown" state.
    • To test this, navigate to System > Servers and Detectors > Overview page and click on the Enforce console refresh button a few times.
    • You should see the Detection server status change with each refresh.
  • You log into your Enforce console, click on a detection server with "Unknown" status, and filereader shows a status of "Running Selected" or "Down"

Depending on the situation, you may see the following errors:

Network Related Issues:

Enforce server Monitor Controller log:

FINE: failed to finish connection to: test:incidentwriter:[IP ADDRESS]:8100

com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information
 .......
Caused by: java.net.ConnectException: Connection timed out: no further information
 ......
FINE: connect operation in SSLConnect failed
com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information
 ........
May 12, 2010 11:57:06 AM com.vontu.communication.transport.ConnectWrapperOperation select
INFO: connectOp select failed

Detection Server:

VontuMonitor log:

INFO   | jvm 12   | 2012/03/22 02:47:04 | WrapperSimpleApp: Encountered an error running main:
INFO   | jvm 12   | 2012/03/22 02:47:04 | WrapperSimpleApp: com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.

BoxMonitor log:

16/03/2012 2:25:17 AM com.vontu.boxmonitor.BoxMonitor main|
SEVERE: (BOXMONITOR.2) BoxMonitor failed to startup
com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.

You may also see:
Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager <init>
SEVERE: failed to init server channel channel manager
Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager SEVERE: failed to init server channel channel manager java.net.BindException: Cannot assign requested address

SSL/Keystore Issues:

On the Detection server the boxmonitor log will show the following:


INFO: Operation com.vontu.communication.transport.AcceptWrapperOperation:1481653881505:null:[email protected] failed with exception: com.vontu.communication.transport.exception.TransportException: javax.net.ssl.SSLHandshakeException: no cipher suites in common
com.vontu.communication.transport.ChannelManager handleOperationFailure
INFO: removing session from cache: [email protected] for id: null

After Adding additional Detection Servers:

The monitor controller log shows:

XXX XX, XXXX XX:XX:XX XX com.vontu.communication.transport.OperationManager enqueue
INFO: blocking on op mgr queue

Filereader status of "Running Selected" or "Down":

FileReader log on the Detection server shows an error: "no disk space" and no additional entries are written to the log.

 

All detection servers show as unknown status in an DLP environment primarily aimed at Discover scanning.

Cause

  • Network Connectivity Issues
  • SSL/Keystore Issues
  • Resource or Tuning Issues
  • Large number of Detection Servers
    • In environments with a large number of Detection servers, it is possible to 'overload' the Enforce queue which controls operations across all transport connections. In instances where this occurs, an 'Unknown' status can be seen across all Detection servers, often proceeding the addition of a new Detection server into the environment.
  • Detection servers switching between 'running' and 'unknown' state:
    • The upgrade process on the Enforce server never completed and is making connections to the Detection servers on the same port used by monitor controller to receive status updates from the Detection servers.
  • Filereader status of "Running Selected" or "Unknown"
    • Detection server disk space is at 100% used.
  • Insufficient maximum java heap memory assigned to the SymantecDetectionServerController process.

Resolution

Network Connectivity Issues:

Follow the steps below to diagnose host unreachable, name resolution and IP filtering issues:

  1. Confirm that the host or fully qualified domain name (FQDN) of the detection server used in the Enforce UI configuration settings
  2. Start a command window (windows) or terminal window (Linux) on the Enforce server.
  3. Ping the IP address of the detection server; use the correct IP for 10.10.10.1
    • ping 10.10.10.1
Possible Results Next Steps
Error message that ping "...could not find host..." There is a name resolution problem. Consult with your Network Administrator to resolve the issue.
100% packet loss (all ping requests timed out) Detection server is not reachable. Correct network and routing issues and repeat the ping test.
Packet loss (some ping requests timed out) Network issues between the Enforce server and detection server. Correct network and routing issues and repeat the ping test.
0% packet loss Continue to the next section.

Use the nslookup command as follows (same for both Windows and Linux):

Example commands

nslookup mon1 

nslookup mon1.some.corp.com

Example Output

Server:
Address:
Name: mon1
Address: 10.10.10.1
Server:
Address:
*** mon1 can't find mon1: Non-existent domain
Possible Results Next Steps
Discrepancy between Name and IP in DNS record, with Enforce Detection Server settings. Should there be a difference in the query result and the server name or IP, contact your DNS Administrator to resolve the issue.

A short term solution is to add the host name or FQDN to the Enforce server local host file.

Windows: c:\windows\system32\drivers\etc\hosts 

Linux: /etc/hosts

Example content:

10.10.10.1    mon1 mon1.some.corp.com   

 

Communication port test

From the Enforce server, use a telnet client to connect to the detection servers default communication port, 8100.

telnet mon1 8100

Possible Results Next Steps
...Could not open connection... Ensure port 8100, inbound from Enforce to Detection Server is allowed through any Network or Host based firewalls.
...Connection refused... Port 8100 may be opened under a different application. Consult with your Systems Administration team to determine if there is a port conflict with another application.
Telnet session Could not open connection to the host, on port 8100: Connect failed At the detection server, ensure the VontuMonitor service is running. Confirm the port 8100 is open and status is reported as LISTENING using the netstat command.

If the above steps do not resolve the issue, try the following:

  1.  On the detection server, open \Protect\Config\Communication.properties in a text editor.
    • By default serverBindName = is commented out. e.g.:
    • # Transport bind address. 
      # The IP address of the NIC that transport server sockets and the registry should be bound to; 
      # The sockets will be bound to the default IP address on the machine if this value is not specified. 
      # serverBindName =
  2. If this is the case, then the Monitor service will bind to the default IP address.
  3. If you see the old IP address entered in the "serverBindName=" setting you can either comment it out or enter the new IP address if you want the Monitor service to bind to that specific IP Address. 
  4. If needed, Update the "serverBindName=" setting.
  5. Restart the Vontu Monitor service on the detection server:
    1. Start > Run > services.msc
    2. Right click the Vontu Monitor services and select Restart.
  6. Now check the Enforce console and you should see the detection server is communicating.

Additional Solutions:

  • Verify there are no Network Access Control Lists (ACLs) restricting permissions to the account used by the DLP services.
    1. On the detection server, Start > Run > services.msc
    2. Right click each of the DLP services and choose properties.
    3. Select the Log On tab.
    4. Note the account being used by the service.
    5. Verify there are no permission restrictions configured in your environment that are applied to this account.
      • For additional assistance with this step please contact your vendor.

------------------------------------------------------------------------------------------

SSL/Keystore Issues:

Each server should have it's own keystore generated when you first add the server to Enforce.

You can verify this by checking the keystore directory on Enforce \SymantecDLP\Protect\keystore for a file that matches this format:

  • enforce..sslKeyStore
  • monitor####.timestamp.sslKeyStore

The number at the end of the monitor name will most often reflect the monitor ID of the the detection server this belongs to.

To identify the monitor ID, mouse-over the detection server in server overview and note the javascript information noted at the bottom of the screen.

The name of the keystore file is case sensitive, so ensure when copying from one OS to another that this is preserved - e.g.,"monitor.timestamp.sslKeyStore".

If not, a Linux Monitor may report the following failure when starting the VontuMonitor service:

Log message Log Resolution

"Unable to resolve the full path of the configuration file, start: No such file or directory"

VontuMonitor.log - Detection server Ensure the protect user has permissions to the /keystore directory on both Enforce and the effected detection server.
Ensure the proper keystore file exists in the /keystore directory.

javax.net.ssl.SSLHandshakeException: no cipher suites in common

VontuMonitor.log - Detection server Check the SSLcipherSuite listing in MonitorController.properties on Enforce and Communication.properties on the detection server have SSL algorithms in common

Resource or Tuning Issues:

Monitor Running out of memory

Within the VontuMonitor log, similar errors to the following error can be seen:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 59136 bytes for Chunk::new
JVM exited unexpectedly.

This is indicative that VontuMonitor does not have enough memory to function properly.

On the Detection server experiencing the issue, navigate to \Protect\config\ and open VontuMonitor.conf
Take note of the following Java Heap Size values:

wrapper.java.initmemory
wrapper.java.maxmemory

By default, these values are set to 128/256 respectively.  When increasing these values, follow these best practice guidelines:

Always increase memory values as a factor of 128 (i.e: 256, 512, 1024).
Always keep the maxmemory value twice the value of initmemory, and neither value should ever be greater than the available RAM on the server.

Restart the Vontu Monitor service.
If the issue continues, restart the Detection server.  This may release any resources being held by the operating system.

Monitor Controller resources are limited

In environments with a large number of Detection servers, it is possible to 'overload' the Enforce queue which controls operations across all transport connections. 

In instances where this occurs, an 'Unknown' status can be seen across all Detection servers, often proceeding the addition of a new Detection server into the environment.

The following steps with increase the transport operation queue size.

NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.

  1. On the Enforce server, navigate to \SymantecDLP\Protect\config.
  2. Open the file, MonitorController.properties, in a text editor.
  3. Locate the configuration key, maxQueueSize. 
    • This setting is 5000 by default, and controls the maximum number of operations that can be pushed out across all transport connections.
  4. Change the value of maxQueueSize to 10000.  

A value this large is enough to handle the number of operation requests seen in large environments, without having a performance impact on memory.

  1. Restart the MonitorController service.

Monitor Controller configured memory limits are too low

The following error is seen in VontuMonitorController.log(Linux Only):

java.lang.OutOfMemoryError: unable to create new native thread

If the out of memory error includes, "unable to create new native thread," this is more likely due to a limit of how many processes a single user can run at one time on RedHad linux.

  1. Log into the Enforce system as root level user.
  2. Switch to root; su - root
  3. Run the following command to print the limits file to the screen: cat /etc/security/limits.conf
  4. Review the limits.d file for any explicit limits to the protect user.

If the protect user is not restricted explicitly, add the following to the limits.conf file to raise the process limit above the default of 1024.

protect soft    nproc   4096
protect hard    nproc   63536

Service Crashing

Within the VontuMonitor.log on the Detection Server, the following error repeats:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate bytes for ChunkPool::allocate
# An error report file with more information is saved as:
# \Protect\bin\hs_err_pidNNNN.log
JVM exited unexpectedly.

3rd-party applications can prevent Java.exe from starting under normal conditions.  It is recommended you collect the hs_err_pidXXX files and contact support.

Large Number Of Detection Servers:

The following steps will increase the transport operation queue size.

NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.

  1. On the Enforce server, navigate to \SymantecDLP\Protect\config.
  2. Open the file, MonitorController.properties, in a text editor.
  3. Locate the configuration key, maxQueueSize.  
  4. This setting is 5000 by default, and controls the maximum number of operations that can be pushed out across all transport connections.
  5. Change the value of maxQueueSize to 30000.  
  6. A value this large is enough to handle the number of operation requests seen in large environments, without having a detrimental impact on memory.
  7. Save and close the file.
  8. Restart the MonitorController service.
  9. All Detection Servers should report in to the Enforce console within a short time period.

Detection servers switching between 'running' and 'unknown' state:

Stop the upgrader process:

  1. Open task manager on the Enforce server:
    1. Start > Run > taskmgr
  2. Find the process named "upgrader-java"
  3. Right-click the "upgrader-java" process and choose "end process".

Filereader status of "Running Selected" or "Unknown"

  1. Check if the disk space on the Detection Server is full:
    1. Log in locally on the Detection server
      • Start > Run > diskmgmt.msc (Windows)
      • df -h or df -a (Linux)
  2. If disk space is at 100%, free up space on the drive.
  3. Restart DLP services on the Detection server.

Insufficient memory allocated to SymantecDetectionServerController process.

It can be observed in DLP environments where the customer is a heavy discover scanning user that runs a large number incremental scans on a daily/weekly basis and also runs a high number of discover scans concurrently a significant amount of java heap memory needs to be allocated to SymantecDLPDetectionServerController process, not doing so may result in the symantecdlpdetectionservercontroller process on Enforce stopping and all detection servers showing as an unknown status. Restarting the service does not resolve the issue and the service is likely to stop again after some time. This may be a silent failure with little or no useful information found in Enforce logs which will pinpoint the issue, the best way to observe the issue is to monitor the memory consumption of the respective DLP processes on the Enforce server and note which ones are running near or at max allocation. Note running a java process near at max allocation will perform poorly and consume excessive CPU due to garbage collection and should be avoided. If process is run near or at max heap allocation the max allocation should be increased giving consideration to available RAM on server and the heap allocation of other DLP services/processes.

Additional information

To learn how to set up and configure DLP detection servers, see Set up and configure detection servers for Data Loss Prevention.