Access logs start unwanted early uploads or small log files are constantly uploading

Products

• ProxySG Software - SGOS

Issue/Introduction

Access logs that start early uploads or that small log files are constantly uploading when not wanted. This occurs although the ProxySG appliance is set to periodically upload log files.

Background

The appliance starts an early upload of the access log(s) if any of the following conditions are true:

• The sum total of the sizes of all the access logs (main, SSL, CIFS, P2P, IM, and so on) reaches the global early upload limit specified in "Start an early upload if total logging reaches xxxx megabytes" in Configuration > Access Logging > General > Global Settings. 
  For example, if the global early upload value is 20000 megabytes and the current SSL log is 19999 megabytes in size, the remaining one megabyte is shared amongst the remaining logs.
• Each log facility has its own early upload threshold. These individual limits are located in Configuration > Access Logging > Logs > General Settings. On the General Settings tab, the console displays log names beside "Log:". In the "Log file limits:" section, the "Start an early upload if log reaches xxxx megabytes" setting is the early upload threshold. 

Examples

• Scenario 1: The global limit is 20GB. The main access log limit is 15GB; all other logs have a limit of 10GB each. If the SSL log is at 9GB and the P2P log is at 8GB, 17 GB of the 20GB total global limit are in use. The main log reaches 3GB, and an early upload occurs.
• Scenario 2: The global limit is 20GB. The main access log has a limit of 15GB; all other logs have a limit of 10GB each. If the SSL log is at 1GB and the P2P log is at 1GB, 2 GB out of the 20GB total limit are in use. The main log size reaches 15GB, and an early upload occurs.

Important: If setting up access logging and configure the main log to be uploaded, be sure to configure other protocols that are intercepted, such as SSL, P2P, and IM. If not, configure them, the main log is uploaded in a regular and timely manner, but the other protocols and logs are not uploaded at all. Over time these other log files grow to the point where the main access log has less room to grow and an early upload occurs because the global early upload limit is hit.

Workaround

In the Management Console, select Configuration > Access Logging > General > Global Settings. Increase the value for "Start an early upload if the log reaches xxxxx megabytes" and save changes. This helps alleviate the problem in the short term, but the problem could recur unless further action is taken. Review the Resolution in this article to permanently alleviate the problem.

Resolution

Before resolution of the issue, determine which logs are backed up and the sizes of those files. Go to https://<ProxySG_IP_address>:8082/Accesslog/statistics and click each of the individual log file. For each, look for AL_STATS_0030 and AL_STATS_0031:

• AL_STATS_0030 is the compressed upload log size (in bytes). 
• AL_STATS_0031 is the uncompressed upload log size (in bytes). 

Record the values for AL_STATS_0030 and AL_STATS_0031 for each log. Then, perform one of the following resolutions.

Resolution 1: Configure upload settings for each log and upload all logs

Perform this resolution if wanting to retain the data collected in the current logs.

1. In the Management Console, select Configuration > Access Logging > Logs > Upload Client.
2. Select a log file.
3. For Upload Client, select an appropriate upload client type such as FTP Client. Then, click Settings and enter the credentials.
   To save space and for efficient uploads, consider using the configured format for uploading files.
4. Click OK.
5. Repeat steps 2 through 4 for all logs.
6. Click Apply to save changes.
7. Select Configuration > Access Logging > General > Global Settings. In the Global Upload section, click Upload All to upload all log files immediately.

Resolution 2: Log only protocols with a configured upload client and delete all logs

Perform this resolution if not wanting to retain the data collected in the current logs.

Step 1:  Change the Default Log to <None> for protocols without an upload client. 

1. In the Management Console, select Configuration > Access Logging > General > Default Logging.
   The tab displays the default logs for each protocol.
2. Locate logs that do not have an upload client configured.
3. Select a log without an upload client and click Edit. On the dialog that opens, select <None> for Default Log.
4. Repeat steps 3 and 4 for other logs without an upload client.
5. Click Apply to save changes.

Step 2: Delete the current logs.

1. Log in to the Command Line Interface (CLI).
2. Enter enable mode and run the following commands:

   Note:  In this example, the SSL protocol is writing data to the SSL log. The commands are for deleting the access logs for the SSL protocol.

   ProxySG#config t
   Enter configuration commands, one per line.  End with CTRL-Z.
   ProxySG#(config)access-log
   ProxySG#(config access-log)edit log ssl
   ProxySG#(config log ssl)commands delete-logs
     ok
   ProxySG#(config log ssl)

3. Issue the commands for all appropriate logs until all of the log files have been deleted. 
4. Go to https://<ProxySG_IP_address>:8082/Accesslog/statistics and verify that the files are deleted.