Configure access logging on ProxySG or ASG to an FTP server, then to Reporter

Products

• Reporter
• ProxySG Software - SGOS

Issue/Introduction

You want to configure access logging for Symantec ProxySG or Advanced Security Gateway (ASG), and then send the log to a Reporter server.

Resolution

This article helps you configure access logging on a ProxySG, upload the files to an FTP server, and then have Reporter process the logs. This article is meant to help in getting access logging and reporting up and running in a relatively short amount of time.

For full details on setting up access logging to Reporter, including other options, see the resources with the URLs below.

Select a Deployment

Perform Initial Configuration

Configure Access Logging

Preamble - Why use FTP?

ProxySG can upload access logs using various protocols. This article focuses on one specific protocol, and that is the FTP protocol.

Symantec recommends the FTP protocol for access log uploads because it offers the best options in case there is ever the need to restore or re-import the access log data. The direct connection (Local Server Source) configuration does not keep the access log data in its raw format, and is recommended for Proof of Concept (POC) only. FTP Server Source is the recommended option for production use . The data is imported into the Symantec Reporter database and the access log file is discarded. With FTP, you can easily create new profiles, recreate profiles, or send data into Symantec Technical Support if need be. The Reporter client (Local Log Source) was a proof-of-concept that was not intended for long-term production use.

Prerequisites

Sizing

Make sure that sizing on Symantec Reporter deployments are appropriate. See the Symantec Reporter Resource Sizing.  Because Reporter is resource intensive (disk, CPU, and memory), for the best performance consider using real hardware and not virtualized hardware.

FTP server

Any proprietary or open source FTP server will work. For simplicity sake this article uses a free open source FTP server named FileZilla Server. Symantec does not implicitly or explicitly promote this free FTP server software. It is merely using it as an example in the configuration of access log to Reporter server setup. Use with discretion when selecting an FTP server.

Note: If interested in connecting to an external FTP server, refer to the steps below.

Steps

The easiest way to set this up is to install the FTP server on a server that has lots of free disk space. After you set up and configure the FTP server, configure ProxySG to upload the access logs to the FTP server.

Next, test connectivity between the FTP server and ProxySG.

Step 1 - Setting up Filezilla FTP server

1. Download and install the Filezilla FTP server.
2. Create a directory where you want the access logs to be stored. For this example, the files are stored in the D:\ftp\proxysg\ directory.
3. In Filezilla, click Edit > Users.
4. On the General page (box on the left-hand side), click Add, under the Users section on the right-hand side. Type in the FTP account name in the pop-up box. In this example, use "proxysg" as the account name. There is no need to make that user a member of a group as the group is optional.
5. Check Enable Account under the account settings section.
6. Check Password, and give the newly created ProxySG user a password. In this example, the password is "symantec". For security purposes, make sure that this password is complex.
7. Click the Shared Folders page. Click Add. Walk the file system directory tree to D:\ftp\proxysg\ and click on the OK button. For files and directories, give that user all file rights (Read, Write, Delete, Append) and all directory rights (Create, Delete, List, + Subdirs) to D:\ftp\proxysg\ . Make sure that D:\ftp\proxysg\ has a capital H next to it. If not, highlight the directory and Click Set as home dir to make that is the home directory for that user. The "H" signifies that D:\ftp\proxysg\ is the home directory for that user. When the ProxySG FTP user logs into the FTP server, the root directory for the ProxySG user will be D:\ftp\proxysg\ . That user will not be able to go any higher in the directory tree.
8. Click OK to save the user.
   
   Note: The "Speed Limits" and "IP Filter" pages are optional and will not be discussed in this article. You can implement them at your own discretion if you want. However, Symantec recommends that you not implement any speed limits or IP filters until after everything is configured and running correctly.
    
9. The Filezilla FTP server should be up and running at this point and your ProxySG user ready to go. 

Step 2 - Setting up access logging on the ProxySG to upload files

1. Log in to the ProxySG Management Console.
2. Click the Default Logging tab, under Configuration > Access Logging > General.
3. Ensure that Enable Access Logging is checked.
4. Click Apply to save your changes.
   
   Important Note:  Make note of all the protocols on the left-hand of the box and the corresponding access logs that they belong to (on the right-hand side). If you want to upload only HTTP traffic and FTP traffic, make sure that all other protocols are set to "". Otherwise there may be problems with early uploads. See Access logs start unwanted early uploads or small log files are constantly uploading for details.
    
5. Make note of the names of the access logs that you want to upload. For all other protocols, set them to "".
6. Click the Global Settings tab (Management Console > Configuration tab > Access Logging > General > Global Settings), and review the defaults on this page.
   
   Note:  Before changing the global early upload limit, read the "Background" information, including examples, in Access logs start unwanted early uploads or small log files are constantly uploading. For now, just leave it at the default settings.
    
7. Click the Upload Client tab, under Configuration > Access Logging > Logs > Upload Client. This is where the access logs are configured to upload their data to the Filezilla server that was set up in Step 1 above. In this example, the HTTP protocol that goes to the main log file is configured. For "Log:", make sure that "main" shows up. When you are doing this for a different log file, make sure that the log file name you want to configure appears here. 
8. For "Upload Client: Client type:", select "FTP Client" and then click Settings.
9. Within "FTP Client settings", enter the IP address of the FTP server. The default FTP port is 21. The Path is "/" (without quotes).
10. Add the username, which in this example is "proxysg".
11. Click Change Primary Password and enter the password twice. In our example, the password is "symantec". Click OK, and then click the Apply.
    
    Note:  If you want to do secure FTP or FTPS between the ProxySG and the Filezilla FTP server, see Uploading ProxySG appliance access logs over FTPS for details.
     
12. In the Upload Client tab, there is a "Save the log file as" setting. To help reduce the amount of disk space used, select the "gzip file" radio button. ProxySG will compress the access log into a .log.gz file name format and upload that to the Filezilla FTP server.
13. Click the Upload Schedule tab next to the Upload Client tab.
    1. Make sure the appropriate log is selected. In this example, it is "main".
    2. In the middle there are two types of uploads. Select "periodically" as the upload type.
    3. At the bottom, under "Upload the log file", you have options to upload the access log on a daily basis at a particular time, or if you want to have it uploaded every so often. See Post installation things to consider at the end of this document, for some additional information regarding what to select here.
14. Test - Now is the time to test. In the "Upload Client" tab, click Test Upload. Go to the Filezilla server. There should be some output stating that the user ProxySG logged in successfully and that it uploaded a file called main_upload_result to the FTP server.

Troubleshooting

If testing from the ProxySG was unsuccessful, troubleshoot the problem as follows:

• Validate the username and password entered in the previous set of steps.
• Double-check the IP address of the FTP server.
• Make sure the Filezilla server is not blocking FTP traffic from an IP subnet.
• Use the Filezilla server interface to view what is happening. The interface can also be configured to show the passwords being sent in clear text so it can verify/validate what is being sent to the FTP server.
• Go to a DOS prompt and open an FTP session from a DOS window to the FTP server. Make sure login using the credentials work and that you can uploading a file to your FTP server works. If login fails, check the FTP user credentials on the FTP server. If login succeeds but upload fails, check the file system permissions and make sure all file and directory permissions have been given. On the Filezilla server, look at the Filezilla server interface.
• From the workstation if you get a long delay (30 - 60 seconds) before receiving an error and are never able to reach the Filezilla server, then there may have a network problem. A network problem can be as simple as a firewall blocking FTP traffic. Or there isn't any route between the workstation and the FTP server.
• If there is a short delay (1 or 2 seconds) before failure, that indicates that the server is reachable, but the port is not open. Make sure Filezilla is running and something like Windows Firewall is not blocking the port.
• Take a packet capture on the ProxySG (Management Console > Maintenance tab > Service Information > Packet Captures) for a minute or so while forcing an access log upload. This should allow visibility to see if the ProxySG is communicating with the FTP server or not. If seeing multiple (three) SYN requests that have no response, then there is probably some sort of networking issue. If seeing SYN > RST three times, then the FTP port is not opened on the remote FTP server, or the wrong FTP port was entered into the access log configuration on the ProxySG.

        9.Repeat steps 4 through 8 above for any other log files that is needed to be uploaded. Make sure that when you set up the log files that the selection of the appropriate log, such as main, or SSL, or P2P, etc...is chosen

Best practices for FTP uploads

• Ensure that your access logs are never left on ProxySG. Monitor your FTP upload/connectivity to ensure the access logs aren't left on ProxySG for days, as this will create a backlog of access logs that need to be uploaded to your FTP server.
• Install a syslog tool that monitors the proxy FTP server, possibly using the second interface so that you can be alerted if the main interface goes down.
• Ensure that you upload your access logs at regular intervals. Find an interval that, on average, uploads a size that is a good fit for your network.

Step 3 - Setting up Reporter

1. Download the latest version of Reporter from support.broadcom.com.
2. Install Reporter. Ensure that the drive where you install to has a lot of available disk space. The install will also ask for an admin user name, password, and license file (not mandatory).
3. From the Reporter server, go to http://127.0.0.1:8081/ in your browser. If remote, you can log in using the IP address of the Reporter server:  http://:8081/. Log in to Reporter using the admin user that was created in the previous step. To view reports in Reporter, a loaded database is required. After creating or loading a database, click View Reports in the top right-hand corner to see the data in the database.  Click OK to remove the message.
4. Within Reporter, under Reporter Settings > Data Settings > Databases, click New. When prompted for a database name, use "proxysg" (example only), and then click Next.
5. Add the source of your log files:
   1. Click New Log Source. A new box will appear asking if you want to pull data from a local file source or an FTP server source. Since the Filezilla FTP server was installed on the Reporter server, select "Local File Source" and click Next. 
   2. Give the log source a name. Call it "proxysg" for this example, and click Next.
   3. For "Directory Path", browse to the FTP directory. In this example, this is "D:\ftp\proxysg\" . Browse to your source directory, and click OK.
   4. For file pattern, leave it a wild card by using * as the wildcard marker, and click Next.
   5. Reporter will ask what to do with the file after processing the log file. Rename, move, or delete it. For this example, use of the default of "Rename: Append '.done' to filename", and click Done.
6. In the Log Sources box, the default polling time is every 10 minutes. Increase or decrease this interval if necessary. Once the polling interval has been selected, click Next.
7. By default, the Reporter server expires data older than 30 days. Increase or decrease the expiration date as required. Also, select when to run the database expiration command. Leave the database at the default setting, and click Next.
   
   About Licensing:  Reporter 9 is licensed based on the number of lines in the Reporter database. Having more data in Reporter server may cause licensing issues. Additionally, Reporter may reach its limit and no longer import access logs into the Reporter database. If Reporter runs, but all the reports contain old data, check the number of lines in Reporter database, comparing it with the Reporter licensing model to ensure that the upper limit has not been reached. If it has reached the limit, expire some data so that Reporter can restart the data import. 
    
8. When prompted for the location of the database files, click Done if the database location is on the hard drive with the most room. The Reporter server starts processing any uploaded access logs, if there are any logs to be processed.
9. Click View Reports (top right-hand corner) to start viewing log data.

Post installation things to consider

Step 2 discusses the frequency of uploads to the FTP server. If the ProxySG is configured for frequent uploads, such as every five minutes, then the FTP server will end up with a lot of small files in that incoming FTP server directory. If the proxy is used in a 24x7 environment, there will be 288 files uploaded to the FTP server on a daily basis. Over a month's time, that will result in approximately 8,600 files, and over a year's time, that will result in about 100,000 files uploaded. File system performance and backup performance can suffer greatly with that many files stored in a single directory. If a Reporter database rebuild needs to occur, all those files will need to be renamed, which can be a time consuming process.

Because of the size and number of files that are uploaded to the FTP user's incoming directory, some sort of periodic movement of files from the FTP user's home directory to a separate storage location may be warranted. For example, a job can be scheduled to kick off a batch file that will move the files from the FTP directory where Reporter looks for new files to another directory. That way a minimal number of files will be maintained. See KB article Access logs management with Reporter for further details.