This is a master troubleshooting article that covers various situations in which one or more Symantec Data Loss Prevention (DLP) Detection servers display an "Unknown" status. For example:
Depending on the situation, you may see the following errors:
Enforce server Monitor Controller log:
FINE: failed to finish connection to: test:incidentwriter:[IP ADDRESS]:8100
com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information
.......
Caused by: java.net.ConnectException: Connection timed out: no further information
......
FINE: connect operation in SSLConnect failed
com.vontu.communication.transport.exception.TransportException: java.net.ConnectException: Connection timed out: no further information
........
May 12, 2010 11:57:06 AM com.vontu.communication.transport.ConnectWrapperOperation select
INFO: connectOp select failed
Detection Server:
VontuMonitor log:
INFO | jvm 12 | 2012/03/22 02:47:04 | WrapperSimpleApp: Encountered an error running main:
INFO | jvm 12 | 2012/03/22 02:47:04 | WrapperSimpleApp: com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.
BoxMonitor log:
16/03/2012 2:25:17 AM com.vontu.boxmonitor.BoxMonitor main|
SEVERE: (BOXMONITOR.2) BoxMonitor failed to startup
com.vontu.boxmonitor.BoxMonitorException: Monitor Error 4162.
You may also see:Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager <init>
SEVERE: failed to init server channel channel manager
Sep 25, 2014 10:11:31 AM com.vontu.communication.transport.ServerChannelManager SEVERE: failed to init server channel channel manager java.net.BindException: Cannot assign requested address
On the Detection server the boxmonitor log will show the following:
INFO: Operation com.vontu.communication.transport.AcceptWrapperOperation:1481653881505:null:com.vontu.communication.transport.SessionIdentifier@708fc9e5 failed with exception: com.vontu.communication.transport.exception.TransportException: javax.net.ssl.SSLHandshakeException: no cipher suites in common
com.vontu.communication.transport.ChannelManager handleOperationFailure
INFO: removing session from cache: com.vontu.communication.transport.SessionIdentifier@708fc9e5 for id: null
The monitor controller log shows:
XXX XX, XXXX XX:XX:XX XX com.vontu.communication.transport.OperationManager enqueue
INFO: blocking on op mgr queue
Resolution increase java heap maxmemory to support the increased detection servers. Java heap maxmemory should be 4 GB per 12 detection servers.
FileReader log on the Detection server shows an error: "no disk space
" and no additional entries are written to the log.
Follow the steps below to diagnose host unreachable, name resolution and IP filtering issues:
ping 10.10.10.1
Possible Results | Next Steps |
---|---|
Error message that ping "...could not find host..." | There is a name resolution problem. Consult with your Network Administrator to resolve the issue. |
100% packet loss (all ping requests timed out) | Detection server is not reachable. Correct network and routing issues and repeat the ping test. |
Packet loss (some ping requests timed out) | Network issues between the Enforce server and detection server. Correct network and routing issues and repeat the ping test. |
0% packet loss | Continue to the next section. |
Example commands
nslookup mon1
nslookup mon1.some.corp.com
Example Output
Possible Results | Next Steps |
---|---|
Discrepancy between Name and IP in DNS record, with Enforce Detection Server settings. | Should there be a difference in the query result and the server name or IP, contact your DNS Administrator to resolve the issue. |
A short term solution is to add the host name or FQDN to the Enforce server local host file.
Windows: c:\windows\system32\drivers\etc\hosts
Linux: /etc/hosts
Example content:
From the Enforce server, use a telnet client to connect to the detection servers default communication port, 8100.
telnet mon1 8100
Possible Results | Next Steps |
---|---|
...Could not open connection... | Ensure port 8100, inbound from Enforce to Detection Server is allowed through any Network or Host based firewalls. |
...Connection refused... | Port 8100 may be opened under a different application. Consult with your Systems Administration team to determine if there is a port conflict with another application. |
Telnet session Could not open connection to the host, on port 8100: Connect failed | At the detection server, ensure the Symantec Detection Server service is running. Confirm the port 8100 is open and status is reported as LISTENING using the netstat command. Note: In the case of network monitor there will be multiple IP addresses, ensure that the IP listening on port 8100 is not the capture card IP address. If it is then update the communication.properties file in the config folder and set the bind address. |
If the above steps do not resolve the issue, try the following:
serverBindName =
is commented out. e.g.:Additional Solutions:
------------------------------------------------------------------------------------------
Each server should have its own keystore generated when you first add the server to Enforce.
You can verify this by checking the keystore directory on Enforce \SymantecDLP\Protect\keystore for a file that matches this format:
The number at the end of the monitor name will most often reflect the monitor ID of the detection server this belongs to.
To identify the monitor ID, mouse-over the detection server in server overview and note the javascript information noted at the bottom of the screen.
The name of the keystore file is case sensitive, so ensure when copying from one OS to another that this is preserved - e.g.,"monitor.timestamp.sslKeyStore".
If not, a Linux Monitor may report the following failure when starting the VontuMonitor service:
Log message | Log | Resolution |
"Unable to resolve the full path of the configuration file, start: No such file or directory" |
VontuMonitor.log - Detection server | Ensure the protect user has permissions to the /keystore directory on both Enforce and the effected detection server. Ensure the proper keystore file exists in the /keystore directory. |
javax.net.ssl.SSLHandshakeException: no cipher suites in common |
VontuMonitor.log - Detection server | Check the SSLcipherSuite listing in MonitorController.properties on Enforce and Communication.properties on the detection server have SSL algorithms in common |
|
boxmonitor log on Network Monitor | Check the SSLcipherSuite listing in MonitorController.properties on Enforce and Communication.properties. Modifying the “SSLautonegotiate” from “false” to “true” |
Within the VontuMonitor log, similar errors to the following error can be seen:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 59136 bytes for Chunk::new
JVM exited unexpectedly.
This is indicative that SymantecDLPDetectionServerService does not have enough memory to function properly.
On the Detection server experiencing the issue, navigate to \Program Files\Symantec\DataLossPrevention\DetectionServer\Services (Windows) or /opt/Symantec/DataLossPrevention/DetectionServer/Services (Linux) and open SymantecDLPDetectionServerController.conf
Take note of the following Java Heap Size values:
wrapper.java.initmemory
wrapper.java.maxmemory
By default, these values are set to 128/256 respectively. When increasing these values, follow these best practice guidelines:
Always increase memory values as a factor of 128 (i.e: 256, 512, 1024).
Always keep the maxmemory value twice the value of initmemory, and neither value should ever be greater than the available RAM on the server.
Restart the SymantecDLPDetectionServerService service.
If the issue continues, restart the Detection server. This may release any resources being held by the operating system.
In environments with a large number of Detection servers, it is possible to 'overload' the Enforce queue which controls operations across all transport connections.
In instances where this occurs, an 'Unknown' status can be seen across some or all Detection servers, often proceeding the addition of a new Detection server into the environment.
The following steps with increase the transport operation queue size.
NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.
A value this large is enough to handle the number of operation requests seen in large environments, without having a performance impact on memory.
The following error is seen in VontuMonitorController.log(Linux Only):
java.lang.OutOfMemoryError: unable to create new native thread
If the out of memory error includes, "unable to create new native thread," this is more likely due to a limit of how many processes a single user can run at one time on RedHad linux.
cat /etc/security/limits.conf
If the protect user is not restricted explicitly, add the following to the limits.conf file to raise the process limit above the default of 1024.
protect soft nproc 4096
protect hard nproc 63536
Within the VontuMonitor.log on the Detection Server, the following error repeats:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate bytes for ChunkPool::allocate
# An error report file with more information is saved as:
# \Protect\bin\hs_err_pidNNNN.log
JVM exited unexpectedly.
3rd-party applications can prevent Java.exe from starting under normal conditions. It is recommended you collect the hs_err_pidXXX files and contact support.
The following steps will increase the transport operation queue size.
NOTE: This change should only have a slight impact on memory as more requests become actively queued for processing.
Stop the upgrader process:
1. If the services on the Detection Server have been restarted before upgrading to the same version, the only solution is to upgrade the Detection Servers to the same version of Enforce to get them to reconnect.
It can be observed in DLP environments where the customer is a heavy discover scanning user that runs a large number of incremental scans on a daily/weekly basis and also runs a high number of discover scans concurrently a significant amount of java heap memory needs to be allocated to SymantecDLPDetectionServerController process, not doing so may result in the SymantecDLPDetectionServerController process on Enforce stopping and all detection servers showing as an unknown status.
Restarting the service does not resolve the issue and the service is likely to stop again after some time. This may be a silent failure with little or no useful information found in Enforce logs that will pinpoint the issue, the best way to observe the issue is to monitor the memory consumption of the respective DLP processes on the Enforce server and note which ones are running near or at max allocation.
Note: Running a java process near max allocation will perform poorly and consume excessive CPU due to garbage collection and should be avoided. If the process is run near or at max heap allocation, the max allocation should be increased giving consideration to available RAM on the server and the heap allocation of other DLP services/processes.