Configure log Retention and Rotation in Siteminder STS

Products

SITEMINDER

Issue/Introduction

Out of the Box configuration for STS on Access Gateway is to rotate the logs daily. It may be necessary to configure log rotation based on other criteria such as size, and limit the number of logs created.

Environment

[SITEMINDER]

Component: Access Gateway

Operating System: ANY

Cause

Siteminder is using Log4J2 for Logging. Log4J2 handles log rotation through a 'Triggering Policy' which defines what conditions initiate a log rollover, and 'Rollover Strategy' which defines how the files are handled during the rollover. Both the Triggering Policy and the Rollover Strategy are configured within the 'RollingFileAppender' section of the 'log4j.xml' file.

Resolution

Siteminder is using Log4J2 for Logging. Log4J2 handles log rotation through a 'Triggering Policy' which defines what conditions initiate a log rollover, and 'Rollover Strategy' which defines how the files are handled during the rollover. Both the Triggering Policy and the Rollover Strategy are configured within the 'RollingFileAppender' section of the 'log4j.xml' file.

The Triggering Policies are contained within the <Policies> element. The <Polcies> element is contained within the <RollingFile> element.

The Rollover Strategy is also contained within the <RollingFile> element. NOT within the <Policies> element, but rather in parallel with it.

<Appenders>
<RollingFile name="SOA_STS_LOG" fileName="D:\\LogFiles\\CA\\secure-proxy\\STS.log" filePattern="D:\\LogFiles\\CA\\secure-proxy\\STS.log.%d{yyyy-MM-dd}" >
<PatternLayout>
<Pattern>%d %-5p [%c] [AgentName=%X{AgentName}] %m%n</Pattern>
</PatternLayout>
<Policies>
<CronTriggeringPolicy schedule="0 0 * * * ?"/>
<TimeBasedTriggeringPolicy interval="1"/>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="20 MB" />
<TimeBasedTriggeringPolicy interval="6" modulate="true"/>
</Policies>
<DefaultRolloverStrategy max="20"/>
</RollingFile>
</Appenders>

===============================
Triggering Policies

CronTriggeringPolicy:

Triggers rollover based on a cron expression. This policy is controlled by a timer and is asynchronous to processing log events, so it is possible that log events from the previous or next time period may appear at the beginning or end of the log file. The filePattern attribute of the Appender should contain a timestamp otherwise the target file will be overwritten on each rollover.

PARAMETERS:

schedule
-> Type: string
-> Description: The cron expression. The expression is the same as what is allowed in the Quartz scheduler. See CronExpression for a full description of the expression.

evaluateOnStartup
-> Type: boolean
-> Description: On startup the cron expression will be evaluated against the file's last modification timestamp. If the cron expression indicates a rollover should have occurred between that time and the current time the file will be immediately rolled over.

SYNTAX:

--------------------
OnStartupTriggeringPolicy:

policy causes a rollover if the log file is older than the current JVM's start time and the minimum file size is met or exceeded.

SYNTAX:

<Policies>
<OnStartupTriggeringPolicy />
</Policies>

--------------------
SizeBasedTriggeringPolicy:

Causes a rollover once the file has reached the specified size. The size can be specified in bytes, with the suffix KB, MB, GB, or TB, for example 20MB. The size may also contain a fractional value such as 1.5 MB. The size is evaluated using the Java root Locale so a period must always be used for the fractional unit. When combined with a time based triggering policy the file pattern must contain a %i otherwise the target file will be overwritten on every rollover as the SizeBased Triggering Policy will not cause the timestamp value in the file name to change. When used without a time based triggering policy the SizeBased Triggering Policy will cause the timestamp value to change.

SYNTAX:

<Policies>
<SizeBasedTriggeringPolicy size="20 MB" />
</Policies>

--------------------
TimeBasedTriggeringPolicy:

Causes a rollover once the date/time pattern no longer applies to the active file. This policy accepts an interval attribute which indicates how frequently the rollover should occur based on the time pattern and a modulate boolean attribute.

PARAMETERS:

interval
-> Type: integer
-> Description: How often a rollover should occur based on the most specific time unit in the date pattern. For example, with a date pattern with hours as the most specific item and increment of 4 rollovers would occur every 4 hours. The default value is 1.

modulate
-> Type: boolean
-> Description: Indicates whether the interval should be adjusted to cause the next rollover to occur on the interval boundary. For example, if the item is hours, the current hour is 3 am and the interval is 4 then the first rollover will occur at 4 am and then next ones will occur at 8 am, noon, 4pm, etc. The default value is false.

maxRandomDelay
-> Type: integer
-> Description: Indicates the maximum number of seconds to randomly delay a rollover. By default, this is 0 which indicates no delay. This setting is useful on servers where multiple applications are configured to rollover log files at the same time and can spread the load of doing so across time.

SYNTAX:

--------------------
CompositeTriggeringPolicy:

Combines multiple triggering policies and returns true if any of the configured policies return true.

===============================
===============================

Rollover Strategies

DefaultRolloverStrategy:

The "DefaultRolloverStrategy" accepts both a date/time pattern and an integer from the filePattern attribute specified on the RollingFileAppender itself. If the date/time pattern is present it will be replaced with the current date and time values. If the pattern contains an integer it will be incremented on each rollover. If the pattern contains both a date/time and integer in the pattern the integer will be incremented until the result of the date/time pattern changes. If the file pattern ends with ".gz", ".zip", ".bz2", ".deflate", ".pack200", ".zst" or ".xz" the resulting archive will be compressed using the compression scheme that matches the suffix.

Parameters:

fileIndex
-> Type: String
-> Description: If set to "max" (the default), files with a higher index will be newer than files with a smaller index. If set to "min", file renaming and the counter will follow the Fixed Window strategy described above.

min
-> Type: integer
-> Description: The minimum value of the counter. The default value is 1.

max
-> Type: integer
-> Description: The maximum value of the counter. Once this values is reached older archives will be deleted on subsequent rollovers. The default value is 7.

compressionLevel
-> Type: integer
-> Description: Sets the compression level, 0-9, where 0 = none, 1 = best speed, through 9 = best compression. Only implemented for ZIP files.

tempCompressedFilePattern
-> Type: String
-> Description: The pattern of the file name of the archived log file during compression.

SYNTAX:

</Policies>
<DefaultRolloverStrategy max="20"/>
</RollingFile>

--------------------
DirectWriteRolloverStrategy:

The "DirectWriteRolloverStrategy" causes log events to be written directly to files represented by the file pattern. With this strategy file renames are not performed. If the size-based triggering policy causes multiple files to be written during the specified time period they will be numbered starting at one and continually incremented until a time-based rollover occurs.

Parameters:

maxFiles
-> Type: String
-> Description: The maximum number of files to allow in the time period matching the file pattern. If the number of files is exceeded the oldest file will be deleted. If specified, the value must be greater than 1. If the value is less than zero or omitted then the number of files will not be limited.

compressionLevel
-> Type: integer
-> Description: Sets the compression level, 0-9, where 0 = none, 1 = best speed, through 9 = best compression. Only implemented for ZIP files.

tempCompressedFilePattern
-> Type: String
-> Description: The pattern of the file name of the archived log file during compression.

SYNTAX

</Policies>
<DirectWriteRolloverStrategy maxFiles="10"/>
</RollingFile>

--------------------

Log Archive Retention Policy: Delete on Rollover:

Log4j-2.5 introduces a Delete action that gives users more control over what files are deleted at rollover time than what was possible with the DefaultRolloverStrategy max attribute. The Delete action lets users configure one or more conditions that select the files to delete relative to a base directory.

DELETE Parameters:

basePath
-> Type: String
-> Description: Required. Base path from where to start scanning for files to delete.

maxDepth
-> Type: int
-> Description:
The maximum number of levels of directories to visit. A value of 0 means that only the starting file (the base path itself) is visited, unless denied by the security manager. A value of Integer.MAX_VALUE indicates that all levels should be visited. The default is 1, meaning only the files in the specified base directory.

followLinks
-> Type: boolean
-> Description: Whether to follow symbolic links. Default is false.

testMode
-> Type: boolean
-> Description: If true, files are not deleted but instead a message is printed to the status logger at INFO level. Use this to do a dry run to test if the configuration works as expected. Default is false.

pathSorter
-> Type: PathSorter
-> Description: A plugin implementing the PathSorter interface to sort the files before selecting the files to delete. The default is to sort most recently modified files first.

pathConditions
-> Type: PathCondition[]
-> Description: Required if no ScriptCondition is specified. One or more PathCondition elements. (See Documentation. Too long to transcribe)
https://logging.apache.org/log4j/2.x/manual/appenders.html

scriptCondition
-> Type: ScriptCondition
-> Description:
Required if no PathConditions are specified. A ScriptCondition element specifying a script.

The ScriptCondition should contain a Script, ScriptRef or ScriptFile element that specifies the logic to be executed. (See also the ScriptFilter documentation for more examples of configuring ScriptFiles and ScriptRefs.)

The script is passed a number of parameters, including a list of paths found under the base path (up to maxDepth) and must return a list with the paths to delete

IfFileName Condition Parameters: (See Documentation. Too long to transcribe)

https://logging.apache.org/log4j/2.x/manual/appenders.html

===============================

SYNTAX

<Appenders>
<RollingFile name="RollingFile" fileName="${baseDir}/app.log"
filePattern="${baseDir}/$${date:yyyy-MM}/app-%d{yyyy-MM-dd-HH}-%i.log.gz">
<PatternLayout pattern="%d %p %c{1.} [%t] %m%n" />
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
<DefaultRolloverStrategy max="100">

<Delete basePath="${baseDir}" maxDepth="2">
<IfFileName glob="*/app-*.log.gz">
<IfLastModified age="P30D">
<IfAny>
<IfAccumulatedFileSize exceeds="100 GB" />
<IfAccumulatedFileCount exceeds="10" />
</IfAny>
</IfLastModified>
</IfFileName>
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>

==============================

These can be configured within the "agent-log4j.xml" file.

Additional Information

Log4J2 Configuration

Log4J2 Appenders