Reporter memory increased due to incorrect access-log naming scheme
Article ID: 204882
Updated On:
Products
Issue/Introduction
Reporter experiences high memory pressure during data processing, resulting in event critical thresholds ALERTS and system events.
Note: The resolution below discusses making a modification to the database configuration file to resolve the Reporter log sorting method when the access-log naming scheme is based on the default log naming scheme. It is important to note, that this article will address the same issue if your logs meet the criteria where Reporter is unable to sort based on time-stamp. Review your naming scheme and make the appropriate adjustments so that the time-stamp in the logs are clear.
Examples would be the following of a Good and Bad naming log scheme. Please read the full article to understand why this is.
Good: Proxy01_main01_20201202162446.log.gz
Bad: Proxy01_main01_20201202_162446.log.gz
Cause
This is due to Reporter incorrectly processing access-log data based on custom access-log naming schemes being created. Reporter, when processing access-log data into the database, builds log-hour and log-day tables. These 'tables' are Reporter's cache memory tables which by default, we store a max of 50 hours into these memory tables. Reporter day tables are larger in size then hour tables.
This can be seen when examining the Reporter preferences.cfg file in data/bluecoat/bcreporter/server/settings
log = {
processing = {
file_descriptors_max_count = "4096"
file_data_set_flush_interval = "1800"
log_table_max_mem_percent = "90"
string_bag_buffer_size = "32768"
stream_data_set_flush_threshold = "3146000"
stream_data_set_flush_interval = "1800"
log_hours_in_memory = {
log_table_min_age = "120"
log_table_min_count = "2"
log_table_target_count = "16"
log_table_max_count = "50" <------
} # log_hours_in_memory
The cause of Reporter needing such a wide time span of tables active in memory is is the result of using a log file naming scheme from the ProxySG which is not very compatible with Reporter's default log file sorting method.
By default, Reporter sorts logs by looking at the filename (starting before the .log.gz) and moving to the left until it finds a non-numeric char, and then sorting by that numeric portion of the string. The ProxySG default file naming scheme is the following naming format: SG_%f_%c_%l%m%d%H%M%S.log.gz such that the most significant 2 numbers in the numeric portion of the filename are the last octet of the ProxySG IP address.
Note: The following from the access-log format is by default and will cause sorting issues; %l = the fourth parameter of the ProxySG appliance IP address (10.102.103.104)
Reference:
%f = log name (facility)
%c = name of the external certificate used for encryption, if any
%l = the fourth parameter of the ProxySG appliance IP address (10.102.103.104)
%m = two-digit month (01 – 12)
%d = two-digit day (01 – 31)
%H = two-digit hour (00 – 23)
%M = two-digit minute (00 – 59)
%S = two-digit second (00 – 59)
.log.gz = extension
This means that instead of sorting log files by time, reporter is first sorting the log files by the ProxySG last octet of their IP address, which causes it to process logs out of time order, and then requires a much wider span of tables to be active in memory simultaneously.
Reporter can handle this different log sorting method, which will properly handle these default naming scheme's and sort them based on time across all ProxySG's in the environment but with a modification.
Resolution
Note: The log ordering method is set per database. These steps would need to be repeated for each database created in the Reporter.
1- Connect to the reporter CLI and stop the reporter process.
2- Edit the database config file for each database.
In each database, you will need to add an additional config line under the top level (under the database_xxxxguid config node). By default, the first two nodes under the database_xxxxguid node will be label and type. Insert a new line after the type line and add this node:
log_ordering_method = "numeric_pre_ext_sg_default_aaammddhhmmss_ignore_ip"
Save the config file changes, then restart the Reporter service. The log processing should now perform better and require less memory.
Detailed step walkthrough: (The steps below are a detailed walkthrough with actual CLI output. It also demonstrates how to use the CLI to get the database guid names to use for the edit process required). ALL EXAMPLES use the example database guid, for example purposes only! Please use your database guid in all commands below.
1 - Use the terminal to access the Reporter CLI. Do not use the serial console.
2 - Enter enable mode. Enter your password when prompted.
3 - Enter # edit-settings list databases
Example CLI output showing the result of the #edit-settings list databases command:
reporter> en
Password:
reporter# edit-settings list databases
database_8dfab3e0291d11eb8076f67c938041b7.cfg
database_c6e1e750041511eb8182f9396d8a41ad.cfg
reporter#
4 - Enter #stop-reporter
This halts the Reporter service (it does not shut down the appliance).
5 - Initiate a file view with the following CLI command: #edit-settings edit databases/<file_name>
Example: Utilizing the output from step 3, reporter# edit-settings list databases, which provided the database guid's, will look like the following below.
reporter# edit-settings edit databases/database_8dfab3e0291d11eb8076f67c938041b7.cfg
Note: (If you make any mistakes and are unable to undo any changes, you can just quit out of the edit mode by using Cntl-C or other methods in vi-edit so you do not save the changes.)
Reporter launches a vi editor. Find the first two nodes under the databse_xxxxguid node that are named label and type. Insert a new line after the type line and add this node:
Example: Below is an example of what the database configuration file should look like once the modification is made to the file. The line is added after the type = "main" entry.
database_8dfab3e0291d11eb8076f67c938041b7 = {
label = "Main_Logs"
type = "main"
log_ordering_method = "numeric_pre_ext_sg_default_aaammddhhmmss_ignore_ip"
state = "enable"
version = ""
creation = "1605649241"
database = {
options = {
database_type = "v9_2"
merge_category = "true"
6 - After you have completed the required edits, you can save the file by hitting ESC to exit the vi-edit insert mode, and then typing the following in the terminal :wq
7 - Enter the following in CLI once the file configuration is saved from the previous command and you are back at the CLI terminal
reporter# edit-settings commit databases/database_8dfab3e0291d11eb8076f67c938041b7.cfg
You can also run the reporter# edit-settings edit databases/database_8dfab3e0291d11eb8076f67c938041b7.cfg command again to reopen the file and verify the config line added was correctly saved to the file.
8 - After you have completed the edits, restart the Reporter service:
reporter# start-reporter
Once Reporter is back-up and running, with these changes the Reporter should start to see a drastic decrease in log hours and days in memory.
Additional Information
Reporter uses vi-editor for modifying Reporter configuration files. Below is an easy quick reference.
VI Editing commands
- i - Insert at cursor (goes into insert mode)
- a - Write after cursor (goes into insert mode)
- A - Write at the end of line (goes into insert mode)
- ESC - Terminate insert mode
- u - Undo last change
- U - Undo all changes to the entire line
- o - Open a new line (goes into insert mode)
- dd - Delete line
- 3dd - Delete 3 lines.
- D - Delete contents of line after the cursor
- C - Delete contents of a line after the cursor and insert new text. Press ESC key to end insertion.
- dw - Delete word
- 4dw - Delete 4 words
- cw - Change word
- x - Delete character at the cursor
- r - Replace character
- R - Overwrite characters from cursor onward
- s - Substitute one character under cursor continue to insert
- S - Substitute entire line and begin to insert at the beginning of the line
- ~ - Change case of individual character
Note: You should be in the "command mode" to execute these commands. VI editor is case-sensitive so make sure you type the commands in the right letter-case.
Make sure you press the right command otherwise you will end up making undesirable changes to the file. You can also enter the insert mode by pressing a, A, o, as required.
Moving within a file
- k - Move cursor up
- j - Move cursor down
- h - Move cursor left
- l - Move cursor right
You need to be in the command mode to move within a file. The default keys for navigation are mentioned below else; You can also use the arrow keys on the keyboard.
Saving and Closing the file
- Shift+zz - Save the file and quit
- :w - Save the file but keep it open
- :q - Quit without saving
- :wq - Save the file and quit
Additional information on manually editing configuration files.
https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/reporter/10-5/Administration_2/Customize_Configurations_10/editconfigs.html
Additional information regarding Reporter log-processing:
https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/reporter/10-5/about_reporter/about_logprcss.html
Feedback
