MSMTC went down with JAVA dump

Products

Mainframe Software Manager (Chorus Software Manager) CHORUS SOFTWARE MANAGER

Issue/Introduction

MSMTC went down with RACF error because the dataset does not have a user alias defined, so it fails when it is trying to be allocated :
IKJ56893I DATA SET MSMUSR.JVM.MSMTC.D200427.T190000.X001 NOT ALLOCATED+

19:00:00 27APR MSMTC 002C S84067 19.00.00 ICH408I USER(msmuser ) GROUP(STCGRP ) NAME(MSM )
19:00:00 27APR MSMTC 002C S84067 19.00.00 CATALOG.MSTTC40 CL(DATASET ) VOL(volser)
19:00:00 27APR MSMTC 002C S84067 19.00.00 INSUFFICIENT ACCESS AUTHORITY
19:00:00 27APR MSMTC 002C S84067 19.00.00 FROM CATALOG.M* (G)
19:00:00 27APR MSMTC 002C S84067 19.00.00 ACCESS INTENT(UPDATE ) ACCESS ALLOWED(READ )
19:00:00 27APR MSMTC 002C S84067 19.00.00 IKJ56893I DATA SET MSMUSR.JVM.MSMTC.D200427.T190000.X001 NOT ALLOCATED+
19:00:00 27APR MSMTC 002C S84067 19.00.00 IGD17102I CATALOG ERROR IN DEFINING NONVSAM DATA SET
19:00:00 27APR MSMTC 002C S84067 19.00.00 MSMUSR.JVM.MSMTC.D200427.T190000.X001
19:00:00 27APR MSMTC 002C S84067 19.00.00 RETURN CODE IS 56 REASON CODE IS 6 IGG0CLFT
19:00:00 27APR MSMTC 002C S84067 19.00.00 IGD306I UNEXPECTED ERROR DURING IGG0CLFT PROCESSING
19:00:00 27APR MSMTC 002C S84067 19.00.00 RETURN CODE 56 REASON CODE 6
19:00:00 27APR MSMTC 002C S84067 19.00.00 THE MODULE THAT DETECTED THE ERROR IS IGDVTSCU
19:00:00 27APR MSMTC 002C S84067 19.00.00 SMS MODULE TRACE BACK - VTSCU VTSCT VTSCR SSIRT

Environment

Release : 6.0

Component : Chorus Software Manager

Cause

JAVA abended and wanted to write a dump and could not due to permission missing.
After defining an alias for msmuser so the MSMUSR.JVM.xxxx dataset / dump could be created and MSMTC failed with these messages:

08.03.35 STC85025 JVMDUMP039I Processing dump event "gpf", detail "" at 2020/04/28 371
371 08:03:35 - please wait.
371
08.03.35 STC85025 JVMDUMP032I JVM requested System dump using 'MSMUSR.JVM.MSMTC.D200428. 372
372 T080335.X&DS' in response to an event
372
08.03.35 STC85025 IGD101I SMS ALLOCATED TO DDNAME (SYS00009) 373
373 DSN (MSMUSR.JVM.MSMTC.D200428.T080335.X001 )
373 STORCLAS (SCNORM) MGMTCLAS (DEFAULT) DATACLAS (DCMVOL)
373 VOL SER NOS= TLRG07
08.03.41 STC85025 IEA827I COMPLETE TRANSACTION DUMP WRITTEN TO 00001 DATASETS 374
374 STARTING FROM 'MSMUSR.JVM.MSMTC.D200428.T080335.X001'
374 PLEASE COMBINE USING COPYDUMP PRIOR TO VIEWING IN IPCS
08.03.41 STC85025 IGD104I MSMUSR.JVM.MSMTC.D200428.T080335.X001 RETAINED, DDNAME=SYS00009
08.03.41 STC85025 JVMDUMP032I JVM requested Java dump using '/u/camsm/msmtmp/javacore.20 376
376 200428.080335.67109329.0002.txt' in response to an event
376
08.03.41 STC85025 JVMDUMP032I JVM requested Snap dump using '/u/camsm/msmtmp/Snap.202004 377
377 28.080335.67109329.0003.trc' in response to an event
377
08.03.41 STC85025 IEA995I SYMPTOM DUMP OUTPUT 378
378 SYSTEM COMPLETION CODE=0C4 REASON CODE=00000010
378 TIME=08.03.41 SEQ=01434 CPU=4002 ASID=003E
378 PSW AT TIME OF ERROR 478D0001 A06429D0 ILC 4 INTC 10
378 NO ACTIVE MODULE FOUND
378 NAME=UNKNOWN
378 DATA AT PSW IS UNAVAILABLE AT THIS TIME
378 GR 0: 00000010 1: 204296D8
378 2: 20DAC710 3: 00000000
378 4: 00000000 5: 00000000
378 6: 00000000 7: 00000000
378 8: 00000000 9: 00000000
378 A: 00000000 B: 00000000
378 C: 00000000 D: 20429F70
378 E: 00FD4740 F: 206429D1
378 END OF SYMPTOM DUMP
08.03.41 STC85025 IEF450I MSMTC MSMTC - ABEND=S0C4 U0000 REASON=00000010 379
379 TIME=08.03.41
08.03.41 STC85025 IEF404I MSMTC - ENDED - TIME=08.03.41
08.03.41 STC85025 $HASP395 MSMTC ENDED - ABEND=S0C4

Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000035
Handler1=000000500871EFB0 Handler2=000000500863F720
gpr0=000000500010AB10 gpr1=00000050082FB300 gpr2=00000050082FB418 gpr3=0000005012136DD0
gpr4=00000050082FB50C gpr5=000000000000003E gpr6=0000005012136E0A gpr7=0000005000F87D00
gpr8=000000001D54B000 gpr9=000000001F50B6D8 gpr10=0000000000000000 gpr11=0000005000100000
gpr12=0000005000108398 gpr13=00000050082FB388 gpr14=0000000020A66F2E gpr15=000000001F50B5A2
fpc=00080000 psw0=0785040180000000 psw1=000000001F50B6A2 sp=00000050082FAA80
bea=000000001F50B676
fpr0=403e000000000000 fpr1=406f000000000000 fpr2=0000000000000000 fpr3=0000000000000000
fpr4=c03dffffffffe800 fpr5=4182666666666667 fpr6=0000000000000000 fpr7=0000000000000000
fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000
fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000
Target=2_90_20200108_436782 (z/OS 02.04.00)
CPU=s390x (3 logical CPUs) (0x2a8d89000 RAM)
----------- Stack Backtrace -----------
protectedIntrospectBacktraceSymbols+0xdf146ea0 (0x0000000000000000)
omrsig_protect+0xdf149b88 (0x0000000000000000)
omrintrospect_backtrace_symbols+0xdf146f90 (0x0000000000000000)
generateDiagnosticFiles+0x835cb840 (0x0000000000000000)
omrsig_protect+0xdf149b88 (0x0000000000000000)
structuredSignalHandlerVM+0x835ca0f0 (0x0000000000000000)
masterSynchSignalHandler+0xdf14d5e8 (0x0000000000000000)
__zerro+0xdf4ec1e0 (0x0000000000000000)
__zerros+0xdf4ec510 (0x0000000000000000)
CEEHDSP+0xdf9d4bb0 (0x0000000000000000)
CEEOSIGJ+0xdf727de0 (0x0000000000000000)
CELQHROD+0xdf9c1278 (0x0000000000000000)
CEEOSIGG+0xdf72ee00 (0x0000000000000000)
CELQHROD+0xdf9c1278 (0x0000000000000000)
open+0xdf599388 (0x0000000000000000)
atoe_open+0xdf206118 (0x0000000000000000)
omrfile_open+0xdf1725b0 (0x0000000000000000)
zip_openZipFile+0x833de170 (0x0000000000000000)
vmizip_openZipFile+0x8355be30 (0x0000000000000000)
initializeClassPathEntry+0x8359c620 (0x0000000000000000)
findLocallyDefinedClass+0x8344c318 (0x0000000000000000)
loadNonArrayClass+0x835e7a80 (0x0000000000000000)
internalFindKnownClass+0x835e40d8 (0x0000000000000000)
initializeRequiredClasses+0x85ae0ea0 (0x0000000000000000)
standardInit+0x85a9d7d0 (0x0000000000000000)
J9VMDllMain+0x85a836f8 (0x0000000000000000)
runJ9VMDllMain+0x835a2718 (0x0000000000000000)
pool_do+0x833c4a40 (0x0000000000000000)
runInitializationStage+0x8359eb48 (0x0000000000000000)
protectedInitializeJavaVM+0x835a3d40 (0x0000000000000000)
omrsig_protect+0xdf149b88 (0x0000000000000000)
initializeJavaVM+0x83597138 (0x0000000000000000)
J9_CreateJavaVM+0x835b25d8 (0x0000000000000000)
JNI_CreateJavaVM+0x832eed58 (0x0000000000000000)
JNI_CreateJavaVM+0xdf213280 (0x0000000000000000)
JzosVM::createJvm()+0xdfbf9e90 (0x0000000000000000)
JzosVM::run(int,char**)+0xdfbfcad8 (0x0000000000000000)
main+0xdfbfd2c8 (0x0000000000000000)
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2020/04/28 08:03:35 - please wait.
JVMDUMP032I JVM requested System dump using 'MSMUSR.JVM.MSMTC.D200428.T080335.X&DS' in response to an event
IEATDUMP in progress with options SDATA=(LPA,GRSQ,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT)
IEATDUMP success for DSN='MSMUSR.JVM.MSMTC.D200428.T080335.X&DS'
JVMDUMP040I System dump written to dataset(s) using name template MSMUSR.JVM.MSMTC.D200428.T080335.X&DS
JVMDUMP032I JVM requested Java dump using '/u/camsm/msmtmp/javacore.20200428.080335.67109329.0002.txt' in response to an event
JVMDUMP010I Java dump written to /u/camsm/msmtmp/javacore.20200428.080335.67109329.0002.txt
JVMDUMP032I JVM requested Snap dump using '/u/camsm/msmtmp/Snap.20200428.080335.67109329.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /u/camsm/msmtmp/Snap.20200428.080335.67109329.0003.trc
JVMDUMP007I JVM Requesting JIT dump using '/u/camsm/msmtmp/jitdump.20200428.080335.67109329.0004.dmp'
JVMDUMP010I JIT dump written to /u/camsm/msmtmp/jitdump.20200428.080335.67109329.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
JVMJZBL1044E The Java virtual machine aborted

Resolution

The IBM Tivoli Asset Discovery (TAD/z) --- USS(x) parameter for its monitoring task - HSIJMON was set to Y the day before.
This caused the JAVA errors, once set back to N, MSMTC came up fine.