Manage log data in Endpoint Protection Manager

Products

• Endpoint Protection

Issue/Introduction

You want to know how to manage log data in Symantec Endpoint Protection Manager (SEPM).

Resolution

You can configure a number of options to manage the logs that are stored in the database.

About log data and storage

The data from all the logs that are uploaded to the console are stored in the console database.

Data is stored in two tables in the database from the following types of logs:

• Application and Device Control logs
• Audit logs
• Enforcer logs
• Network Threat Protection logs
• System logs

The data from other logs is stored in a single table. You can set the log options for managing the database logs that are stored in two tables.

The single table that contains the other logs' data is managed by using the database maintenance options in the site properties. You can set the database maintenance options that affect the data that is stored in a single table.  For the logs that are stored in two tables, one table (table A) is the current log table. New log entries are written into this table. When the log threshold or expiration occurs, new log entries are stored in the second table (table B). The data remains in table A until table B reaches its threshold or the number of days that is specified in the Expired after field. At that time, table A is cleared completely and new entries are stored there. The information in table B remains until the switch occurs. Switching from one table to the other, also called sweeping the logs from the database, occurs automatically. The timing of the switch depends on the log settings that you set in the site properties. The process is the same regardless of whether the sweep is automatic or manual.

You can perform a manual log sweep after backing up the database, if you prefer to use this method as part of routine database maintenance.

If you allow an automatic sweep to occur, you may lose some log data if your database backups do not occur frequently enough. If you regularly perform a manual log sweep after you have performed a database backup, it ensures that you retain all your log data. This procedure is very useful if you must retain your logs for a relatively long period of time, such as a year.

Note: The manual procedure that is described below does not affect the data in the logs that are stored in a single table in the database.

Sweeping log data from the database manually

You can manually clear the logs, but this procedure is optional and you do not have to do it.

To manually sweep log data from the database

1. To prevent an automatic sweep of the database until after a backup occurs, increase the Site Properties Log Settings to their maximums.
2. Perform the backup, as appropriate.
3. On the computer where the manager is installed, open a Web browser and type the following URL:
   https://localhost:8443/servlet/ConsoleServlet?ActionType=ConfigServer&action=SweepLogs
   After you have performed this task, the log entries for all types of logs are saved in the alternate database table. The original table is kept until the next sweep is initiated.
4. To empty all but the most current entries, perform a second sweep. The original table is cleared and entries then start to be stored there again.

Log data from legacy clients

The Symantec Endpoint Protection reporting functions use a temporary folder, drive:\Program Files(x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Temp, for several purposes. Some administrators may want to schedule their own automated tasks to periodically clean this temporary folder.

Configuring log settings for the servers in a site

To help control disk space usage, you can configure the number of entries that are kept on the server in a site's logs. You can also configure the number of days the entries are kept. You can configure different settings for the different sites.

Note: Log information on the console Logs tab on the Monitors page is presented in logical groups for you to view. The log names on the Site Properties Log Settings tab correspond to log content rather than to log types on the Monitors page Logs tab.

For a description of each configurable option, you can click Tell me more for that type of report on the console. Tell me more displays the context-sensitive help.

To configure log settings for the servers in a site

1. In the console, click Admin.
2. On the lower left, click Servers.
3. Select the database of the site that you want to configure.
4. Under Tasks, click Edit Database Properties.
5. On the Log Settings tab, set the number of entries and number of days to keep log entries for each type of log.  You can set sizes for management server logs, client logs, and Enforcer logs.
6. Click OK.

About configuring event aggregation

You configure event aggregation for client logs in two locations on the console.

Location Description

Use this location to configure the aggregation for risk events. The default aggregation time is 5 minutes. The first occurrence of an event is immediately logged. Subsequent occurrences of the same events are aggregated and the number of occurrences is logged on the client every 5 minutes.

On the Policies page, Antivirus and Antispyware policy, Miscellaneous, Log Handling tab

Use this location to configure the aggregation of Network Threat Protection events. Events are held on the clients for the damper period before they are aggregated into a single event and then uploaded to the console. The damper period helps to reduce events to a manageable number. The default damper period setting is Auto (Automatic). The damper idle period determines the amount of time that must pass between log entries before the next occurrence is considered a new entry. The default damper idle is 10 seconds. On the Clients page, Policies page, Client Log Settings

Configuring client log settings

If you have installed Symantec Endpoint Protection, you can configure some client log options. You can configure the number of entries kept in the logs and the number of days that each entry is kept on the client. You can configure settings for the following client logs:

• Control
• Packet
• Risk
• Security
• System
• Traffic

For the Security, Risk, and Traffic logs, you can also configure the damper period and the damper idle period to be used for event aggregation. You can configure whether or not to upload each type of client log to the server, and the maximum size of the uploads.

If you choose not to upload the client logs, it has the following consequences:

• You cannot view the client log data from the console by using the Logs tab on the Monitoring pane.
• You cannot back up the client logs when you back up the database.
• You cannot export the client log data to a file or a centralized log server.

To configure client log settings

1. On the console, click Clients.
2. On the Policies tab, under Location-independent Policies and Settings, under Settings, click Client Log Settings.
3. In the Client Log Settings for group name dialog box, set the maximum file size and the number of days to keep log entries.
4. Check Upload to management server for any logs that you want the clients to forward to the server.
5. For the Security log and Traffic log, set the damper period and the damper idle period. These settings determine how frequently Network Threat Protection events are aggregated.
6. Set the maximum number of entries that you want a client to upload to the manager at a time.
7. Click OK.

About configuring client log handling options for antivirus and antispyware policies

You can configure the following log handling options for antivirus and antispyware policies:

• Which antivirus and antispyware events are forwarded from clients to the Antivirus and Antispyware Protection logs on the server
• How long the events in the Antivirus and Antispyware Protection logs are retained on the server
• How frequently aggregated events are uploaded from clients to the server

Backing up the logs for a site

Log data is not backed up unless you configure Symantec Endpoint Protection to back it up. If you do not back up the logs, then only your log configuration options are saved during a backup. You can use the backup to restore your database, but the logs in the database are empty of data when they are restored. This configuration option is located with the other backup options for local sites on the Servers page of the Admin page. You can choose to keep up to ten versions of site backups. You should ensure that you have adequate disk space to keep all your data if you choose to keep multiple versions.

To back up the logs for a site

1. On the console, click Admin.
2. Select a database server.
3. Under Tasks, click Edit Backup Settings.
4. In the Backup Settings group box, check Back up logs.
5. Click OK.

About uploading large amounts of client log data

If you have a large number of clients, you may have a large volume of client log data.

You should consider whether or not you want to reduce the volume of data by using the following configurations:

• Upload only some of the client logs to the server.
• Filter the less important risk events and system events out so that less data is forwarded to the server.

If you still plan to upload very large amounts of client log data to a server, you need to consider the following factors:

• The number of clients in your network
• The heartbeat frequency, which controls how often the client logs are uploaded to the server
• The amount of space in the directory where the log data is stored before being inserted into the database

A configuration that uploads a large volume of client log data to the server at frequent intervals can cause space problems. If you must upload a large volume of client log data, you may have to adjust some default values to avoid these space problems. As you deploy to clients, you should monitor the space on the server in the log insertion directory and adjust these values as needed. The default directory where the logs are converted to .dat files and then written into the database is drive:\Program Files(x86)\Symantec\Symantec Endpoint Protection Manager\data\inbox\log. The location of the server data directory is set during installation when you are asked to select the server data folder. You can run the Management Server Configuration Wizard from the Start menu to change this directory if desired. The \inbox\log directory is automatically added to the directory you set.

The frequency with which the client logs are uploaded is configured on the Policies page of the Clients page, under Communications Settings. The default frequency is to upload the logs every five minutes. 

To adjust the values that control the space available on the server, you must change these values in the registry. The registry keys that you need to change are located on the server in:

• HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Symantec\Symantec Endpoint Protection\SEPM.

About managing log events in the database

The database receives and stores a constant flow of entries into its log files. You must manage the data that are stored in the database so that the stored data does not consume all the available disk space. Too much data can cause the computer on which the database runs to crash.

You should understand your default database maintenance settings and change them if the disk space that the database uses seems to grow constantly. If there is a large spike in risk activity, you may need to delete some data to protect the available disk space on the server.

Configuring database maintenance options for logs

Administrators can configure database maintenance options for the data that are stored in the logs. Database maintenance options help you to manage the size of your database by specifying compression settings and how long to keep data.

For information about the specific database maintenance options, refer to the context-sensitive help on the Site Properties for site name dialog box Database tab.

To configure database maintenance options for logs

1. On the console, click Admin.
2. Click the site's database, and under Tasks, click Edit Database Properties.
3. On the Log Settings tab*, under Risk Log Settings, set the following options:
   1. Set the number of days to keep risk events. To retain the subset of risk infection events after the threshold that you set for risk events, check the Do not delete infection events check box.
   2. Set how frequently you want to compress identical risk found events into a single event.
   3. Set the number of days to keep the events that have been compressed. This value includes the time before the events were compressed. For example, suppose that you specify to delete compressed events after ten days and specify to compress events after seven days. In this case, the events are deleted three days after they are compressed.
   4. Set the number of days to keep acknowledged and unacknowledged notifications.
   5. Set the number of days to keep scan events.
   6. Set the number of days to keep commands that you have run from the console and their associated command status information. After this time, Symantec Endpoint Protection can no longer distribute the commands to their intended recipients.
   7. Check the check boxes if you want to delete unused virus definitions and the virus events that contain EICAR as the name of the virus. The EICAR test virus is a text file that the European Institute for Computer Anti-Virus Research (EICAR) developed. It provides an easy and safe way to test most antivirus software. You can download it from the EICAR Web site. You can use it to verify that the antivirus portion of Symantec Endpoint Protection works.
4. Click OK to save your changes.

About using the Interactive SQL utility with the embedded database

If you choose to use the embedded database with Symantec Endpoint Protection or Symantec Network Access Control, you should note the following information. When you run the database application named Interactive SQL (dbisqlc.exe), it blocks the insertion of data into the embedded database. If you use the application for a while, .dat files accumulate in the drive:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\inbox\log directories.

To alleviate the buildup of the .dat files and restart data insertion into the database, close the application.

Changing timeout parameters

If database errors occur when you view reports or logs that contain a lot of data,
you can make the following changes:

• Change the Microsoft SQL server connection timeout
• Change the Microsoft SQL server command timeout

The reporting defaults for these values are as follows:

• Connection timeout is 300 seconds (5 minutes)
• Command timeout is 300 seconds (5 minutes)

If you get CGI or terminated process errors, you might want to change other timeout parameters. See the Symantec Knowledge Base article called "Reporting server does not report or shows a timeout error message when querying large amounts of data."

To change timeout parameters

1. Open the Reporter.php file, which is located in the drive:\Program Files(x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Resources directory. 
2. Use any text editor to add the following settings to the file:

* $CommandTimeout = ####
* $ConnectionTimeout = ####

Timeout values are in seconds. If you specify zero, or leave the fields blank, the default settings are used.

About recovering a corrupted client System Log on 64-bit computers

If the System Log becomes corrupted on a 64-bit client, you may see an unspecified error message in the system logs on the console. If corrupted, you cannot view the data in the log on the client and the data does not upload to the console. This condition can affect data in the console Computer Status, Risk, and Scan logs and reports.

To correct this condition, you can delete the corrupted log file and the serialize.dat file on the client. These files are located on the client in Drive:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\Logs\AV\date.Log. After you delete these files, the log file is recreated and begins to log entries correctly.

Technical Information

When running this command in a browser window:

https://localhost:8443/servlet/ConsoleServlet?ActionType=ConfigServer&action=SweepLogs

On screen, the following message is displayed:

<?xml version="1.0" encoding="UTF-8"?>
<Response ResponseCode="0"/>

This is the expected output. It is not an error.

With SEP 14, add the following to the conf.properies file, to allow the command to run.

scm.configserver.allowed.actions=SweepLogs

To confirm that action has been taken, examine the server logs in the Admin, Servers section of the GUI console. There should be an entry listed similar to:

May 21, 20xx 11:22:59 AM BST: Some logs have been swept. [Site: Site Your_Site_Name] [Server: Your_Server_Name]
May 21, 20xx 12:13:04 PM BST: Some logs have been swept. [Site: Site Your_Site_Name] [Server: Your_Server_Name]