SPLUNK not receiving as much access logs, from the Proxy, as expected from their SPLUNK collector

Products

• ISG Proxy
• ProxySG Software - SGOS

Issue/Introduction

It's been reported that the SPLUNK reported is not receiving the access logs expected from their SPLUNK collector. 

Only 500 are seen in the Splunk database compared to 100,000 in Symantec Reporter. 

The issue appears to be with the log string details in this doc - What is the recommended Splunk access log format that can be used for the proxy access log? (broadcom.com)

If you tail the logs on the ProxySG appliances, you only see a couple of logs every second. you see many more when viewing other log formats

SPLUNK Log format – A couple of lines are shown every second when viewing the logs continuously.
BC Reporter format – Many logs are seen every couple of seconds when viewing the logs continuously, as expected.

Test Log (not pointing to any log server)
Using SPLUNK format – Only seeing a couple of lines every second.
Using other pre-defined log formats – Many logs as expected.

CPU and Memory are fine on the appliances, so it's not like they are struggling. 

Apart from the ridiculous log string size being advised by SPLUNK, what other reasons could be causing this?

Note: The issue described with the SPLUNK log format on the ProxySG appliances seems to be affecting the volume of logs collected, resulting in fewer logs being sent to Splunk compared to Symantec Reporter.

Environment

SG/ASG/ISG-Proxy

Cause

Here are some potential causes and considerations to troubleshoot and resolve the problem:

1. Log Format Complexity and Processing
   
   
   • Complex Log Strings: If the SPLUNK log format is overly complex or verbose, it could be causing delays in log processing. The appliances may take longer to format each log entry, resulting in fewer logs being generated and sent per second.
     
     
   • Consideration: Simplify the log format, if possible. Reducing the number of fields or the verbosity of the log entries can help increase the rate at which logs are processed and sent.
     
     For the recommended log format, refer to the SPLUNK doc. with URL:
     
     Configure logging in your Blue Coat ProxySG appliance for the Splunk Add-on;
     
     Additional Docs.: Recommended Splunk access log format that can be used for the proxy access log & Sending Access Logs to a Syslog server
     
     Please, ensure to use only this recommended format. Also ensure to have implemented the access log upload, following the recommended procedure, as shared, and using syslog.
2. Log Filtering and Sampling
   
   
   • Filters: Check if there are any filters applied to the logs being sent to Splunk. Filters can restrict the number of logs sent based on certain criteria, such as log severity or specific fields.
     
     
   • Sampling: Ensure that log sampling isn't configured, which might limit the number of logs being sent to conserve resources.
     
     Please note that none of the above is a recommendation with the Edge SWG (ProxySG), and is shouldn't be done in SPLUNK either, for access log upload to SPLUNK.
     
     
3. Data Truncation or Parsing Issues
   
   
   • Truncation: If the log entries are being truncated due to length limitations in the SPLUNK format, this could lead to incomplete logs or errors in log transmission.
     
     
   • Parsing Errors: Misconfigured or incorrect log formats can cause parsing errors, leading to logs being dropped or not recognized by Splunk.
     
     
4. Network or Communication Issues
   
   
   • Bandwidth: Ensure there is sufficient network bandwidth between the ProxySG appliances and the Splunk server. Network congestion or bandwidth limitations can result in logs being delayed or dropped.
     
     
   • Protocol Issues: Verify that the correct protocol and settings are being used for log transmission. Issues with encryption, authentication, or other network settings can interfere with log delivery. Ensure "syslog" protocol is used for the access log transmission.
     
     For further problem isolation, you may implement sending event logs from the Edge SWG (ProxySG) to SPLUNK, to see how much logs are sent, compared with the access logging. Ref.: Enable Syslog on the Edge SWG (ProxySG). With the SGAC, the "Maintenance" feature has been replaced with the "Administration" feature. Where the same behavior is observed, this would point to, possibly, other environmental factors, or the SPLUNK itself. If the log transmission is significantly more, for event logging, this would further point to complexity, or verbose issues, with the access log format implemented for the log transmission to SPLUNK.
     
     Collecting a PCAP, in real time, would help show whether there network/communication issues. Collect the PCAP from the Edge SWG (ProxySG), with filters set to the IP address of the syslog server (SPLUNK) and the port number implemented in the access log upload client settings. You may share the PCAP on a Technical Support ticket, for validation checks.
     
     
5. Splunk Configuration and Performance
   
   
   • Indexer Performance: Check the performance and configuration of the Splunk indexers. Overloaded or misconfigured indexers may not be able to keep up with the log volume, causing logs to be dropped or delayed.
     
     
   • Queue Management: Review Splunk’s queue settings (such as input, parsing, and indexing queues) to ensure they are appropriately sized for the expected log volume.
     
     The above, if not kept in check, can lead to buffer/TCPZeroWindow issues, on the side of the SPLUNK, forcing it to take only as much access log, as has been reported.
     
     
6. ProxySG Logging Settings
   
   
   • Log Level: Ensure that the log level on the ProxySG appliances is set to capture all necessary events. A lower log level might result in fewer logs being generated.
     
     
   • Log Rotation and Storage: Check if logs are being rotated or archived too quickly, which might prevent them from being sent to Splunk in time. Ref.: Rotate access logs 

Resolution

Steps to Troubleshoot and Resolve:

1. Review and Simplify Log Format: Work with your team to simplify the SPLUNK log format. Ensure it captures all necessary information without unnecessary complexity.
   
   
2. Check Filters and Sampling: Confirm that there are no filters or sampling configurations that are limiting the log output.
   
   
3. Inspect Splunk Configuration: Review the Splunk system’s configuration, focusing on network settings, indexer performance, and queue management.
   
   
4. Monitor Network Health: Verify that the network path between the ProxySG appliances and the Splunk server is healthy and free of bottlenecks.
   
   
5. Test with Alternative Formats: If possible, test using different log formats to see if the issue persists, which can help isolate the problem to either the format itself or other factors.

By addressing these areas, you should be able to identify and rectify the factors limiting the log transmission to Splunk, ensuring a more comprehensive log collection process.