Detection Server stuck on Starting state

Products

• Data Loss Prevention Network Monitor and Prevent for Email

Issue/Introduction

Detection servers are stuck at starting state. Log samples:

 

RequestProcessor1.log:

com.vontu.mta.rp.connection.ConnectionsStateManager1G connectToMessageChains
INFO: RPT(32): Unable to get a connection to the message chain.  Will try again in 300 milliseconds.

FileReader0.log

com.symantec.dlp.utilities.filesystem.filelocker.FileLocker lockFiles
WARNING: Failed to lock file: ...........................................................................................................................................................................................................................LegacyEndpointDocSource.9592.437.rdx in directory: /var/Symantec/DataLossPrevention/DetectionServer/<version>/../../ServerPlatformCommon/<version>/index
java.io.IOException: Failed to lock file and create hard link: ...........................................................................................................................................................................................................................LegacyEndpointDocSource.9592.437.rdx
 at com.symantec.dlp.utilities.filesystem.filelocker.HardLinkOSFileLock.lock(HardLinkOSFileLock.java:71)
 at com.symantec.dlp.utilities.filesystem.filelocker.FileLocker.lockFiles(FileLocker.java:141)
 at com.symantec.dlp.utilities.filesystem.filelocker.FileLocker.lockFiles(FileLocker.java:181)
 at com.vontu.detection.policy.engines.config.DefaultIndexLoaderFactory.createIndexLoader(DefaultIndexLoaderFactory.java:169)
 at com.vontu.detection.policy.engines.config.DefaultIndexLoaderFactory.createIndexLoader(DefaultIndexLoaderFactory.java:99)
 at com.vontu.detection.config.DefaultDetectionEngineConfiguration.createDocumentIndexEngine(DefaultDetectionEngineConfiguration.java:371)
 at com.vontu.detection.config.DefaultDetectionEngineConfiguration.initialize(DefaultDetectionEngineConfiguration.java:309)
 at com.vontu.messaging.FileReader.createDetectionEngineConfiguration(FileReader.java:629)
 at com.vontu.messaging.FileReader.createMessageChainFactories(FileReader.java:516)
 at com.vontu.messaging.FileReader.createMessageChainPool(FileReader.java:501)
 at com.vontu.messaging.FileReader.start(FileReader.java:430)
 at com.vontu.messaging.FileReaderSetup.initialize(FileReaderSetup.java:105)
 at com.vontu.messaging.FileReader.main(FileReader.java:302)
Caused by: java.nio.file.FileSystemException: /var/Symantec/DataLossPrevention/DetectionServer/<version>/../../ServerPlatformCommon/15.7/index/............................................................................................................................................................................................................................LegacyEndpointDocSource.9592.437.rdx -> /var/Symantec/DataLossPrevention/DetectionServer/<version>/../../ServerPlatformCommon/15.7/index/...........................................................................................................................................................................................................................LegacyEndpointDocSource.9592.437.rdx: File name too long
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
 at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
 at java.nio.file.Files.createLink(Files.java:1086)
 at com.symantec.dlp.utilities.filesystem.filelocker.HardLinkOSFileLock.createHardLinkToFile(HardLinkOSFileLock.java:128)
 at com.symantec.dlp.utilities.filesystem.filelocker.HardLinkOSFileLock.lock(HardLinkOSFileLock.java:66)
 ... 12 more

 

Resolution

Below solution copied from closed case

For remediation to restore production:

1. Stop DLPDetectionServerService on affected system.
2. It’s safe to remove the . files in the index directory (default /var/Symantec/DataLossPrevention/ServerPlatformCommon/15.8.00000/index) – command rm -f .*
3. Start DLPDetectionServerService – check for any 2920 errors, as below:
4. If no 2920’s exist, then database profile is loaded successfully.  If there is a db profile issue that remains, copy missing .rdx files from Enforce or from another detection server with  SymantecDLPDetectionServer stopped, then start services and re-test.