Parsing and analyzing Access Logs for troubleshooting

book

Article ID: 171483

calendar_today

Updated On:

Products

ProxySG Software - SGOS

Issue/Introduction

The purpose of this article is to provide steps to extract Access Logs from the ProxySG and order them in a spreadsheet so that it can be analyzed further in search for issues.

Resolution

  1. Retrieve the Access Logs by sending them from the ProxySG to an SR (recommended method), from an FTP server or from the Advanced URL /Accesslog/directory
  2. Open the Access Log and remove "#Fields:" located in the 5th line so that the line begins with "date" instead. Doing this will ease the parsing later on.
  3. Copy the content of the file
  4. Open an Excel spreadsheet and paste the content on the first cell of the spreadsheet. All the fields will appear in the first column.
  5. In order to split the fields into columns, we need to select the whole column then go to Data > Text to Columns > Delimited > Next > Space > Next > Finish (tested in Excel 2016). This will align the field name (first row) with each value below it.
  6. 5. (Optional) If sorting via a specific field is required (such as domain name), a filter can be created by selecting the top cell in the desired column ("cs-host" log field) and go to Data > Filter (tested in Excel 2016). A dropdown menu will appear in the cell we selected. After this, we can select the domain/s we want to inspect. By doing this, we can see all the requests being made to a given domain at a given time.

 

Other possible uses for this include but are not limited to:

  • Checking HTTP response codes obtained ("sc-status" column)
  • Viewing User Agents currently processed by the proxy ("cs(User-Agent)" column)
  • Looking at the categorizations the proxy performed on the sites ("cs-categories" column)