OPSMVS shutdown tasks failed at system shutdown/ 0C4 in UNKNOWN

Products

• OPS/MVS Event Management & Automation

Issue/Introduction

CICS Regions were not in the system but automation had them in stopping status, changed to down status.  An ADABAS task had to be canceled.

The only thing I find in common with these CICS regions is an abend after CICS says termination is complete:

 IEA995I SYMPTOM DUMP OUTPUT  874

SYSTEM COMPLETION CODE=0C4  REASON CODE=00000004

TIME=00.06.51  SEQ=11569  CPU=0000  ASID=021F

PSW AT TIME OF ERROR  478C4000   9A2FAB5C  ILC 4  INTC 04

   NO ACTIVE MODULE FOUND - PRIMARY NOT EQUAL TO HOME

   NAME=UNKNOWN

   DATA AT PSW  1A2FAB56 - B038BF12  B028B219  10009180

   AR/GR 0: 00000000/00288800   1: 00000000/00288000

         2: 00000000/45113108   3: 00000000/00000000

         4: 01FF000A/00288800   5: 01FF000A/00000800

         6: 00000000/9871B64A   7: 00000000/000000F8

         8: 00000000/00000000   9: 00000000/7F4E8000

         A: 00000000/1875F000   B: 01FF000A/45112D24

         C: 00000000/1A2FA9E0   D: 00000000/45112DA0

         E: 00000000/9871B6B2   F: 00000000/1A2FA9E0

END OF SYMPTOM DUMP

IEA838I SYSMDUMP SUPPRESSED AS A DUPLICATE OF:  898

        ORIGINAL:DATE 21286 TIME 23:27:28:05 CPU 0037AD373906

        MOD/CSVEXPR CSECT/CSVEXPR PIDS/5752SC1CJ AB/S00C4

        REXN/CSVEXPR FI/B038BF12B028B21910009180 REGS/0F17C

        REGS/0C17C HRC1/00000004

IEF404I UKCICT15 - ENDED - TIME=00.06.52

$HASP395 UKCICT15 ENDED - RC=0000

 

CSVEXPR is a contents supervisor module that deals with system exits.  The module mentioned in the message isn’t necessarily the one that ABENDed but could be its recovery that got control for an earlier ABEND from someone else and decided to capture a dump. 

HMVS     22196 00:06:52.28 STC27849 00000090  IEA838I SYSMDUMP SUPPRESSED AS A DUPLICATE OF: 867           

                                867 00000090          ORIGINAL:DATE 21286 TIME 23:27:28:05 CPU 0037AD373906

                                867 00000090          MOD/CSVEXPR CSECT/CSVEXPR PIDS/5752SC1CJ AB/S00C4    

                                867 00000090          REXN/CSVEXPR FI/B038BF12B028B21910009180 REGS/0F17C  

                                867 00000090          REGS/0C17C HRC1/00000004                             

 

Just before this, I see that OPS/MVS exit OPUSPREX was deactivated because it had taken an ABEND0C4:

 

HMVS     22196 00:06:52.14 STC27864 00000090  CSV430I MODULE OPUSPREX FOR EXIT BPX_PREPROC_TERM HAS BEEN MADE

                                              INACTIVE DUE TO ABEND=0C4000 REASON=00000004                  

 

  Unfortunately a logrec record was not cut for this and the dump was suppressed by DAE so I can’t say for sure what happened but it looks like OPS/MVS uses exit point BPX_PREPROC_TERM to be alerted for UNIX processes terminating so this might be why OPS stopped shutting things down.  I set DAE to capture the next dump so if we see it again we’ll have doc.

Environment

Release : 13.5

Component : OPS/MVS

Cause

Process Blocks became depleted during the time period in question

Resolution

The problem was caused by a process block depletion.  The shortage existed from about 00:06:51 - 00:06:55.   

It is recommended to spot check of how many rules were firing on the different events.  Some messages had up to 8 rules firing, some of which were not justified and should be eliminated.  An example is the OPS3490O with 4 rules.  2 of those 4 appear to be suppressing this message, and by default, suffixed O messages only go to a log.  Look for duplication of efforts against a single rule in the time period of the problem, as well as just prior to.  

By simplifying the rules that are firing on the various messages, there can be significantly less risk of hitting the process block problem.