On Symantec Security Information Manager v4.5.x prior to installing MR -2 IBM DB2 crashes and fills the / partition.

Products

• Security Information Manager

Issue/Introduction

Symptoms
Notification on Symantec Security Information Manager v4.5.x reporting that the disk is full and the appliance stops responding

Cause

The root cause is identified in IBM DB2 defect JR24995: HANDLE LEAK IN GET_DB_CONFIG MAY LEAD TO SQL0969N (SQL ERROR -99999). and addressed in MR-2

Resolution

You can apply Symantec Security Information Manager v4.5 MR-2 or following the steps below for the hardware you are using:

Preventative Steps for the Symantec Security Information Manager v4.5 9600 appliance
The following steps should be performed on a weekly basis to prevent the failure. Users will not be able to login to the SSIM Appliance and event processing will stop while these steps are executed.

1. Check the disk usage by executing the following commands as the “db2admin” user:

$du –h /dblog/sesa

The reported result should be under 3.5 GB.

$ls –l /dblog/sesa/*.LOG | wc –l

The reported result should be under 75.

2. If you don’t see the expected results listed above, please restart the database by executing the following commands as the “db2admin” user:
$db2 force applications all

$db2 terminate

$db2stop

$db2start

3. Check the disk usage again by repeating step 1 above. If you still don’t see the expected results, please follow the steps in the next section “Recovering from the Disk Full Condition.”

Recovering from the Disk Full Condition
The following steps should be performed if the SSIM Appliance is already in a disk full state.

1. Stop all services on the SSIM Appliance by executing the following command as “root” user:

# /opt/Symantec/simserver/simdbmu/bin/sh stopall.sh

You can check the status of the services by running the following command as “root” user:

# status

All services listed above should be in the “OFF” state.

2. Stop the DB2 database server by executing the following commands as “db2admin” user:
$db2 force applications all

$db2 terminate

$db2stop

3. Check the timestamp of the DB2 transaction logs by executing the following command as “db2admin” user:
$ls –l /dblog/sesa/*.LOG

Delete the oldest DB2 transaction logs under the /dblog/sesa directory until there are only 75 logs left.

4. Restart the DB2 database server by executing the following command as “db2admin” user:
$db2start
5. Copy the cr-munger-tmp-tables.sql script to the /tmp directory of the appliance.
6. Recreate the default munger temporary tables by executing the following command as “db2admin” user:
$dos2unix /tmp/cr-munger-tmp-tables.sql

$db2 –tf /tmp/cr-munger-tmp-tables.sql

*Note: This will not recreate any custom mungers.

7. Start all services on the SSIM Appliance by executing the following commands as “root” user:
# /opt/Symantec/simserver/simdbmu/bin/sh startall.sh

You can check the status of the services by running the following command as “root” user:

# status

All services listed above should be in the “UP” state.

Preventative Steps for the Symantec Security Information Manager v4.5 9550 appliance

1. Lower the limit by updating log4j files on the appliance using an ssh connection or the DRAC (not via UI).
Copy /opt/Symantec/simserver/eventfinder-log4j.properties to log4j.properties because eventfinder reads the setting from log4j.properties.

Lower the MaxFileSize in all log4j to 10MB using the following command to find all the log4j.properties files you need to change:
As root run below command to see the current settings in the files and which ones to change

# find /opt/Symantec -name "*log4j*" -exec egrep -i MaxFileSize {} /dev/null \;

Lower the MaxBackupIndex in all log4j.properties file you find to 1 or 2MB.
As root run below command to see the current settings in the files and which ones to change:

# find /opt/Symantec -name "*log4j*" -exec egrep -i MaxBackup {} /dev/null \;

Lower the log level to WARN except DBMU.
As root run below command to see the current settings in the files and which ones to change

# find /opt/Symantec -name "*log4j*" -exec egrep rootLogger {} /dev/null \;

2. DB2 logging - lower the limit of number of DB2 logs by updating DB2 parameter (current limit is 60 for 9500). Usually about 60~70 log files with total size of 2.5G~3G are allocated
under /dblog/sesa and they don't grow because DB2 creates new logs and removes old logs to keep the limit.
As db2admin run below command to lower the limit to 50 (2G~2.5G), run db2 command as below.

$ db2 update db cfg for sesa using LOGPRIMARY 50

3. Reboot the appliance after the above change.

4. Check SSIM appliance disk space daily using the df command. If disk space is running out, do a manual purge. Without above change, only 7.5G becomes available in / partition after the installation. As root type these commands:

#df -h
#find / -name "*.log*" -print | xargs ls -l |sort -r -n -k 5,5
#find / -name "*.out*" -print | xargs ls -l |sort -r -n -k 5,5

References
This has been identified as a problem in IBM’s DB2 Database server with the following APAR:

APAR number: JR24996

http://www-1.ibm.com/support/docview.wss?uid=swg1JR24996