Implemented Script Lookup Plugin and Enforce is throwing errors now.
Checking netstat there are a lot of sockets being held open by java processes.
Error in logs is:
com.vontu.command.CommandException: Unable to write to syslog: host=127.XX.0.XXX, port=514com.vontu.command.CommandException: Unable to write to syslog: host=127.XX.0.XXX, port=1xx
at com.vontu.manager.command.systemevent.SyslogLogger.execute(SyslogLogger.java:184)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:844)
at com.vontu.command.CommandRuntime.executeInstructions(CommandRuntime.java:827)
at com.vontu.command.CommandRuntime.executeCommands(CommandRuntime.java:719)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:675)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:695)
at com.vontu.command.CommandRuntime.triggerFired(CommandRuntime.java:978)
at com.vontu.command.CommandTrigger.fireTriggerEvent(CommandTrigger.java:116)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvent(SystemEventCommandTrigger.java:136)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvents(SystemEventCommandTrigger.java:195)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.performWork(SystemEventCommandTrigger.java:252)
at com.vontu.util.concurrent.WorkerThread.run(WorkerThread.java:84)
Caused by: com.vontu.util.syslog.SyslogException: Can't create sending socket.
at com.vontu.util.syslog.Syslog.initSockets(Syslog.java:149)
at com.vontu.util.syslog.Syslog.<init>(Syslog.java:74)
at com.vontu.manager.command.systemevent.SyslogLogger.execute(SyslogLogger.java:143)
... 11 more
Caused by: java.net.SocketException: maximum number of DatagramSockets reached
at sun.net.ResourceManager.beforeUdpCreate(ResourceManager.java:73)
at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:77)
at java.net.DatagramSocket.createImpl(DatagramSocket.java:337)
at java.net.DatagramSocket.<init>(DatagramSocket.java:239)
at java.net.DatagramSocket.<init>(DatagramSocket.java:196)
at com.vontu.util.syslog.Syslog.initSockets(Syslog.java:140)
... 13 more
com.vontu.util.syslog.SyslogException: Can't create sending socket. com.vontu.util.syslog.SyslogException: Can't create sending socket.
at com.vontu.util.syslog.Syslog.initSockets(Syslog.java:149)
at com.vontu.util.syslog.Syslog.<init>(Syslog.java:74)
at com.vontu.manager.command.systemevent.SyslogLogger.execute(SyslogLogger.java:143)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:844)
at com.vontu.command.CommandRuntime.executeInstructions(CommandRuntime.java:827)
at com.vontu.command.CommandRuntime.executeCommands(CommandRuntime.java:719)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:675)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:695)
at com.vontu.command.CommandRuntime.triggerFired(CommandRuntime.java:978)
at com.vontu.command.CommandTrigger.fireTriggerEvent(CommandTrigger.java:116)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvent(SystemEventCommandTrigger.java:136)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvents(SystemEventCommandTrigger.java:195)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.performWork(SystemEventCommandTrigger.java:252)
at com.vontu.util.concurrent.WorkerThread.run(WorkerThread.java:84)
Caused by: java.net.SocketException: maximum number of DatagramSockets reached
at sun.net.ResourceManager.beforeUdpCreate(ResourceManager.java:73)
at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:77)
at java.net.DatagramSocket.createImpl(DatagramSocket.java:337)
at java.net.DatagramSocket.<init>(DatagramSocket.java:239)
at java.net.DatagramSocket.<init>(DatagramSocket.java:196)
at com.vontu.util.syslog.Syslog.initSockets(Syslog.java:140)
... 13 more
java.net.SocketException: maximum number of DatagramSockets reachedjava.net.SocketException: maximum number of DatagramSockets reached
at sun.net.ResourceManager.beforeUdpCreate(ResourceManager.java:73)
at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:77)
at java.net.DatagramSocket.createImpl(DatagramSocket.java:337)
at java.net.DatagramSocket.<init>(DatagramSocket.java:239)
at java.net.DatagramSocket.<init>(DatagramSocket.java:196)
at com.vontu.util.syslog.Syslog.initSockets(Syslog.java:140)
at com.vontu.util.syslog.Syslog.<init>(Syslog.java:74)
at com.vontu.manager.command.systemevent.SyslogLogger.execute(SyslogLogger.java:143)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:844)
at com.vontu.command.CommandRuntime.executeInstructions(CommandRuntime.java:827)
at com.vontu.command.CommandRuntime.executeCommands(CommandRuntime.java:719)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:675)
at com.vontu.command.CommandRuntime.execute(CommandRuntime.java:695)
at com.vontu.command.CommandRuntime.triggerFired(CommandRuntime.java:978)
at com.vontu.command.CommandTrigger.fireTriggerEvent(CommandTrigger.java:116)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvent(SystemEventCommandTrigger.java:136)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.fireTriggerEvents(SystemEventCommandTrigger.java:195)
at com.vontu.manager.command.systemevent.SystemEventCommandTrigger.performWork(SystemEventCommandTrigger.java:252)
at com.vontu.util.concurrent.WorkerThread.run(WorkerThread.java:84)
There is no way to change the DatagramSockets limit.
The only configuration option that we have in relation to java would be the Java heap memory settings.
For the Script Lookup Plugin, Increasing the threadpool from 5-20 in the Plugin.Properties file and increasing the Java Heap memory for the Incident Persister process resolved the Datagram sockets error.