Gateway 9.2 CR 7
2 nodes Primary/Secondary
Only primary is in the Load Balancer
[Analysis]
************
Collected good DCT before test
Collected DCT after server unresponsive - packet trace
Analysed/compare data
Look to disable the log sink options to send to a remote syslog. At least do this as a test
Thread Name
SyslogMessageSender-UDP-syslogvhi.MYCOMPANY.COM/10.0.0.1:514
State
Blocked
The above is from the thread dump
Disable Oracle Audit - error occurs all throughout the log
Unable to get jdbc connection datasource: Oracle_Audit_ReadOnly
Collected another DCT - packet trace
Second data with Oracle out of picture and syslog - same pattern
SSG log threadID 8067 locks/consume cycles of the gateway process
2019-02-19T13:05:19.784-0600 WARNING 8067 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -5: Request Headers: accept:*/*,
2019-02-19T13:06:56.389-0600 WARNING 8067 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -5: Request Cookies: _
2019-02-19T13:07:58.392-0600 WARNING 8067 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -5: Response Headers:
2019-02-19T13:07:58.392-0600 WARNING 8067 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -5: Response Cookies:
2019-02-19T13:07:58.393-0600 WARNING 8067 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -5: Request routed to: https://SERVER.MYCOMPANY.COM:443/private/payment/list?customerId=999999999999999
2019-02-19T13:07:58.393-0600 WARNING 8067 com.l7tech.server.message: Message processed successfully
DCT performs separate backtrace dumps of the gateway process 20 seconds apart
Dump 1 isolate thread 8067
"tomcat-exec-executor-408" #8067 daemon prio=5 os_prio=0 tid=0x00007f0530214000 nid=0x5bcf runnable [0x00007f0499b2f000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
- locked <0x00000005248ec1e0> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.util.logging.StreamHandler.flush(StreamHandler.java:259)
- eliminated <0x00000005248ec140> (a com.l7tech.server.log.ConsoleMessageSink$L7ConsoleHandler)
at com.l7tech.server.log.ConsoleMessageSink$L7ConsoleHandler.publish(Unknown Source)
- locked <0x00000005248ec140> (a com.l7tech.server.log.ConsoleMessageSink$L7ConsoleHandler)
.
.
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000007b0c2f000> (a java.util.concurrent.ThreadPoolExecutor$Worker)