Fragmented Log Files on Log server every few days

Products

• ProxySG Software - SGOS

Issue/Introduction

Help explain why the ProxySG appliance which pushes its access logs via SCP to a log server fragments the files every few days. 
The max individual log size is 10GB and early upload is 10gb as well. Log upload schedule is daily at midnight periodically with 60 seconds delay between connections. 

Please find below the created file names and info for the last weeks: 


-rw-r----- 1 yyyyyyy yyyyyyy  944496433 xxx xx 00:02 Logs-xxxxxxxxx-Proxy-20230729.gz
-rw-r----- 1 yyyyyyy yyyyyyy  855062790 xxx xx 00:01 Logs-xxxxxxxxx-Proxy-20230730.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3754107167 xxx  x 00:06 Logs-xxxxxxxxx-Proxy-20230731.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3756019206 xxx  x 00:06 Logs-xxxxxxxxx-Proxy-20230801.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3870037426 xxx  x 00:06 Logs-xxxxxxxxx-Proxy-20230802.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3741226977 xxx  x 00:07 Logs-xxxxxxxxx-Proxy-20230803.gz
-rw-r----- 1 yyyyyyy yyyyyyy      32799 xxx  x 00:00 Logs-xxxxxxxxx-Proxy-20230804-6.gz
-rw-r----- 1 yyyyyyy yyyyyyy 2928863252 xxx  x 00:04 Logs-xxxxxxxxx-Proxy-20230804.gz
-rw-r----- 1 yyyyyyy yyyyyyy      21898 xxx  x 00:00 Logs-xxxxxxxxx-Proxy-20230805-6.gz
-rw-r----- 1 yyyyyyy yyyyyyy  842700675 xxx  x 00:01 Logs-xxxxxxxxx-Proxy-20230805.gz
-rw-r----- 1 yyyyyyy yyyyyyy        257 xxx  x 00:00 Logs-xxxxxxxxx-Proxy-20230806-6.gz
-rw-r----- 1 yyyyyyy yyyyyyy  913853556 xxx  x 00:01 Logs-xxxxxxxxx-Proxy-20230806.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3681336646 xxx  x 00:06 Logs-xxxxxxxxx-Proxy-20230807.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3790781594 xxx  x 00:07 Logs-xxxxxxxxx-Proxy-20230808.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3807933049 xxx xx 00:06 Logs-xxxxxxxxx-Proxy-20230809.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3659278167 xxx xx 00:05 Logs-xxxxxxxxx-Proxy-20230810.gz
-rw-r----- 1 yyyyyyy yyyyyyy 2868196750 xxx xx 00:05 Logs-xxxxxxxxx-Proxy-20230811.gz
-rw-r----- 1 yyyyyyy yyyyyyy  859325521 xxx xx 00:01 Logs-xxxxxxxxx-Proxy-20230812.gz
-rw-r----- 1 yyyyyyy yyyyyyy  846897483 xxx xx 00:01 Logs-xxxxxxxxx-Proxy-20230813.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3705177796 xxx xx 00:06 Logs-xxxxxxxxx-Proxy-20230814.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3974220346 xxx xx 00:07 Logs-xxxxxxxxx-Proxy-20230815.gz
-rw-r----- 1 yyyyyyy yyyyyyy 3770580665 xxx xx 00:07 Logs-xxxxxxxxx-Proxy-20230816.gz

Cause

Access log files from a ProxySG appliance or similar proxy server can become fragmented over days due to several factors, including:

Log Rotation Configuration: Many proxy servers are configured to rotate log files at regular intervals, such as daily or weekly. When log rotation occurs, the current log file is closed, and a new one is created. This can result in multiple log files spanning several days.

Log Volume: The volume of log data generated by the proxy server can be substantial, especially in large networks. To manage the size of log files, log rotation is essential.

Disk Space Constraints: To prevent log files from consuming all available disk space, proxy servers typically implement log rotation to create new log files while archiving or compressing older ones.

Retention Policies: Organizations may have policies in place that specify how long log data should be retained. Log files are often archived or deleted based on these policies.

Maintenance and Backup: During maintenance or backup operations, log files may be temporarily inaccessible, leading to fragmentation if the proxy server continues to generate log data during that time.

Restart or Failover: Proxy servers may be restarted or undergo failover procedures for various reasons. When this happens, log files may be closed and new ones created, causing fragmentation.

Log Analysis and Reporting: Organizations may use log analysis and reporting tools that expect log data to be split into manageable files. Log rotation facilitates the processing of log data.

Resolution

To address log file fragmentation and manage log data effectively, you should consider the following:

• Implement a log rotation strategy that aligns with your organization's requirements and policies. 

  Check the ProxySG's log rotation settings. Log rotation is a common practice to split logs into manageable files. Ensure that the log rotation settings are configured appropriately. You can usually adjust these settings in the ProxySG's management interface.

  Note: Log rotation is only available for the 'continuous' upload type. 

  Ref.: Rotate Access Logs

• Use log management and analysis tools to consolidate log data from fragmented log files into a central repository for easier analysis and reporting. Symantec Reporter is one of such access log management and analysis tool primarily designed for analyzing and reporting on web traffic and web security events. It is part of Symantec's suite of web security solutions and is often used in conjunction with Symantec ProxySG (Proxy Secure Gateway) appliances.

• Regularly review and adjust log retention policies to ensure compliance with regulatory requirements and operational needs. Please refer to the Tech. Article below.
  
  Ref.: Delete access logs on Edge-SWG (formerly ProxySG)
  
  

• Monitor disk space usage to prevent log files from consuming too much storage. For more, please refer to the Tech. Article below.
  
  Ref.: Edge SWG (Formerly ProxySG) Access log failed to upload to a remote server

By managing log files effectively and implementing appropriate log rotation and retention policies, you can ensure that log data remains accessible, organized, and compliant with your needs and policies.