Configuring API Gateway to Avoid Auditing/Processing Shutdown, Caused by Audits Exceeding Storage Threshold.
search cancel

Configuring API Gateway to Avoid Auditing/Processing Shutdown, Caused by Audits Exceeding Storage Threshold.

book

Article ID: 8534

calendar_today

Updated On:

Products

CA API Gateway

Issue/Introduction

API Gateway stops auditing and/or processing requests.

  • Observed in /var/log/mysqld.log

170901 21:04:57 [ERROR] /usr/sbin/mysqld: The table '#sql-977_90eda7' is full
170908 21:05:22 [ERROR] /usr/sbin/mysqld: The table '#sql-977_93d076' is full
170909 21:05:29 [ERROR] /usr/sbin/mysqld: The table '#sql-977_943866' is full
170914 21:06:34 [ERROR] /usr/sbin/mysqld: The table '#sql-977_966c1e' is full

  • Observed in /opt/SecureSpan/Gateway/node/default/var/logs/ssg_0_0.log

2017-09-15T16:27:37.132-0400 WARNING 19485495 com.l7tech.server.audit.AuditArchiver: 2205: Audit Archiver error: Receiver not enabled.
2017-09-15T16:27:37.132-0400 WARNING 618 com.l7tech.server.message: Message was not processed: Undefined (-1)
2017-09-15T16:27:37.133-0400 WARNING 618 com.l7tech.server.audit.AuditRecordManagerImpl: Unable to save AuditRecord: Database is full!
2017-09-15T16:27:37.255-0400 WARNING 822 com.l7tech.server.SoapMessageProcessingServlet: Message processing suspended by the Audit Archiver
2017-09-15T16:27:37.257-0400 WARNING 822 com.l7tech.server.MessageProcessor: 4: Message processing suspended: Audit records database disk usage exceeded emergency limit (90 >= 90). Exception caught!

Environment

- CA API Gateway, all form factors.- Audits stored internally in MySQL Database.

Cause

API Gateway stops auditing and/or processing requests, due to audits going over the set storage threshold and triggering the archiver shutdown policy.

  • Once the amount of space taken by audit records in the database reaches the threshold defined in audit.archiverShutdownThreshold Cluster-wide Property, Gateway will stop auditing and/or processing requests, based on the strategy outlined in the audit.managementStrategy Cluster-wide Property.
  • Default value for the threshold defined in audit.archiverShutdownThreshold Cluster-wide Property is %90 of the smaller value between the size of the logical volume holding the Database file, and the maximum size for the Database file in MySQL settings.
  • Default value for audit.managementStrategy Cluster-wide Property is set to 'STOP', which results in the gateway stopping processing requests and terminating audit logging. 
  • By default, database file is stored under /var/lib/mysql/ibdata, on the /dev/mapper/vg00-lv_db logical volume.
  • Maximum size for the Database file in MySQL settings is defined in /etc/my.cnf
    • innodb_data_file_path=ibdata:100M:autoextend:max:15851M

Resolution

Fix

  • In the case where maximum database size in MySQL settings is smaller that the size of the logical volume hosting the Database file, increasing value for autoextend:max in /etc/my.cnf, and restarting MySQL and SSG services result in the gateway resuming processing of requests and logging auditing.
  • Doing a clean-up on the database through removing audit records, as described in  KB42833, and restarting MySQL and SSG services result in the gateway resuming the processing of requests and logging auditing.

Prevention

  • Changing audit.managementStrategy Cluster-wide Property from the default value of 'STOP' to 'BYPASS' results in the gateway continuing to process requests, but terminating audit logging. Internal Gateway logging continues, with a SEVERE-level message that audit logging has stopped.
  • Installation and scheduling of an audit record maintenance script, as described in KB42480, would keep the quantity of audit records stored in the database on an appropriate level for the environment.
  • Configuration of a FTP Audit Archiver, as described in the documentation, will result in audits being backed up externally, and removed from the database.
  • Monitoring disk utilization of Gateway logical volumes, as described in KB10795, could provide customers with early warning on Gateway's database filling up the hosting logical volume, due to storage of audit records.

Additional Information

Using instructions provided in KB article 111628 , the information obtained could be used to determine how to manage the removal of older audit records:

  • The size of the Databases in the MySQL instance
  • The amount of data per table
  • The newest and oldest audit record
  • The number of audit records it contains
  • The date range of these records
Powered by Wolken Software