CICS Regions were not in the system but automation had them in stopping status, changed to down status. An ADABAS task had to be canceled.
The only thing I find in common with these CICS regions is an abend after CICS says termination is complete:
IEA995I SYMPTOM DUMP OUTPUT 874
SYSTEM COMPLETION CODE=0C4 REASON CODE=00000004
TIME=00.06.51 SEQ=11569 CPU=0000 ASID=021F
PSW AT TIME OF ERROR 478C4000 9A2FAB5C ILC 4 INTC 04
NO ACTIVE MODULE FOUND - PRIMARY NOT EQUAL TO HOME
NAME=UNKNOWN
DATA AT PSW 1A2FAB56 - B038BF12 B028B219 10009180
AR/GR 0: 00000000/00288800 1: 00000000/00288000
2: 00000000/45113108 3: 00000000/00000000
4: 01FF000A/00288800 5: 01FF000A/00000800
6: 00000000/9871B64A 7: 00000000/000000F8
8: 00000000/00000000 9: 00000000/7F4E8000
A: 00000000/1875F000 B: 01FF000A/45112D24
C: 00000000/1A2FA9E0 D: 00000000/45112DA0
E: 00000000/9871B6B2 F: 00000000/1A2FA9E0
END OF SYMPTOM DUMP
IEA838I SYSMDUMP SUPPRESSED AS A DUPLICATE OF: 898
ORIGINAL:DATE 21286 TIME 23:27:28:05 CPU 0037AD373906
MOD/CSVEXPR CSECT/CSVEXPR PIDS/5752SC1CJ AB/S00C4
REXN/CSVEXPR FI/B038BF12B028B21910009180 REGS/0F17C
REGS/0C17C HRC1/00000004
IEF404I UKCICT15 - ENDED - TIME=00.06.52
$HASP395 UKCICT15 ENDED - RC=0000
CSVEXPR is a contents supervisor module that deals with system exits. The module mentioned in the message isn’t necessarily the one that ABENDed but could be its recovery that got control for an earlier ABEND from someone else and decided to capture a dump.
HMVS 22196 00:06:52.28 STC27849 00000090 IEA838I SYSMDUMP SUPPRESSED AS A DUPLICATE OF: 867
867 00000090 ORIGINAL:DATE 21286 TIME 23:27:28:05 CPU 0037AD373906
867 00000090 MOD/CSVEXPR CSECT/CSVEXPR PIDS/5752SC1CJ AB/S00C4
867 00000090 REXN/CSVEXPR FI/B038BF12B028B21910009180 REGS/0F17C
867 00000090 REGS/0C17C HRC1/00000004
Just before this, I see that OPS/MVS exit OPUSPREX was deactivated because it had taken an ABEND0C4:
HMVS 22196 00:06:52.14 STC27864 00000090 CSV430I MODULE OPUSPREX FOR EXIT BPX_PREPROC_TERM HAS BEEN MADE
INACTIVE DUE TO ABEND=0C4000 REASON=00000004
Unfortunately a logrec record was not cut for this and the dump was suppressed by DAE so I can’t say for sure what happened but it looks like OPS/MVS uses exit point BPX_PREPROC_TERM to be alerted for UNIX processes terminating so this might be why OPS stopped shutting things down. I set DAE to capture the next dump so if we see it again we’ll have doc.
OPS/MVS
Process Blocks became depleted during the time period in question
The problem was caused by a process block depletion. The shortage existed from about 00:06:51 - 00:06:55.
It is recommended to spot check of how many rules were firing on the different events. Some messages had up to 8 rules firing, some of which were not justified and should be eliminated. An example is the OPS3490O with 4 rules. 2 of those 4 appear to be suppressing this message, and by default, suffixed O messages only go to a log. Look for duplication of efforts against a single rule in the time period of the problem, as well as just prior to.
By simplifying the rules that are firing on the various messages, there can be significantly less risk of hitting the process block problem.