Scheduler DB Tablespace was full the DB shutdown

Products

• DSERIES- SERVER
• CA Workload Automation DE - System Agent (dSeries)

Issue/Introduction

Scheduler Oracle DB shutdown. Now getting 911 error when every jobs try to run even after a cold start.

Environment

CA Product = WORKLOAD AUTOMATION DE
Product Code = DSWKLA99000
Product Release =11.3 SP3
Component Code = DSRIES

Cause

After analyzing the logs we could see the below exception :

20180404 16:15:01.348 [essential] [INFO] Thread-25: New StatusMessage: [date: Wed Apr 04 16:15:01 EDT 2018, type: Log message, loggerName: relationaldatabase, level: ERROR, threadName: DM.Appl.MD_TPMS_DELETION_SCRIPTS_RT_QUEUE.1, message: SQL Exception for query: INSERT INTO ESP_APPLICATION (APPL_ID, JOB_NAME, APPL_NAME, APPL_GEN_NO, START_DATE_TIME, END_DATE_TIME, SCHEDULED_DATE_TIME, STATE, TAG, COMMENTS, EVENT_NAME) VALUES (?,?,?,?,?,?,?,?,?,?,?); the exception is: ORA-00001: unique constraint (DSERIES.SYS_C007144) violated 
, stackTrace: [java.sql.SQLException: ORA-00001: unique constraint (DSERIES.SYS_C007144) violated, , 

Resolution

From the exception we could figure out that the problem is due to the duplicate keys which resulted in the unique key constraint exception. We maintain the individual tables max keys in the esp_long_keys which will be used to determine the next max key for a table. We could see there is some difference in the individual tables and the esp_longkeys, due to which it couldn't determine the key properly. After updating the max ids properly, the server started well.