as_server (application server) or event_demon (scheduler) crash with ORA-24550

Products

• CA Workload Automation AE - Business Agents (AutoSys)
• CA Workload Automation AE - System Agent (AutoSys)
• CA Workload Automation AE - Scheduler (AutoSys)
• Workload Automation Agent
• Autosys Workload Automation

Issue/Introduction

The application server crashes during its operations, as_server log shows the following:

ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=128] [si_int=0] [si_ptr=(nil)] [si_addr=(nil)]
kpedbg_dmp_stack()+400<-kpeDbgCrash()+210<-kpeDbgSignalHandler()+121<-skgesig_sigactionHandler()+272<-__sighandler()<-__lll_unlock_elision()+48<-_ZN7ispUtil11MutexUnlockEPv()+14

Environment

Release : 12.0

Component : CA Workload Automation AE (AutoSys)

Cause

Real error message is being supprssed by the Oracle Signal handler.  It needs to be disabled first so we can capture the correct stack from the application server, analyze/identify root cause based on the core dump that can then be generated correctly by the crashing process.  Similar crash could happen on scheduler too

Resolution

1) Make a backup of TNS_ADMIN\sqlnet.ora file and then add the following setting to the sqlnet.ora file:

DIAG_SIGHANDLER_ENABLED=FALSE  

2) Save the file

3) Restart the application server (or scheduler, if scheduler was the one crashing)

4) Now a proper core dump could be generated and analyzed (  https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/autosys-workload-automation/12-0/troubleshooting/troubleshooting-ae/capture-diagnostic-information.html#:~:text=kb/241215.-,UNIX%20or%20Linux%3A,-Obtain%20the%20core )

 

DIAG_SIGHANDLER_ENABLED=TRUE turns off the Oracle diagnostics, the diagnostic error from the crashing process is being suppressed.  So we needed it to be turned off to investigate this further.