XCOM started task XCOMM0053E abend S306 and hung

Products

• XCOM Data Transport - z/OS
• XCOM Data Transport

Issue/Introduction

In a Production environment the XCOM for z/OS started task abended with S306 and hung. See the following error messages:

11.39.05 STC15284  IDI0034I Fault analysis skipped due to: EXCLUDE option specification (FAST)                                       
 11.39.06 STC15284  CSV019I REQUESTED MODULE CWDDFLTP NOT ACCESSED, IS IN NON-APF LIBRARY/CONCATENATION                               
 11.39.06 STC15284  CSV028I ABEND306-0C  JOBNAME=XCOM      STEPNAME=SERVER                                                            
 11.39.06 STC15284  AB5AA- SNAP5A ESTAE(X) RECOVERY WAS ENTERED                                                                       
 11.39.06 STC15284  AB5C3- PSW = 07041000 80000000 00000000 013F6812, IL = 02, INT = 000D                                             
 11.39.06 STC15284  AB5C4- 64-BIT GPR VALUES:                                                                                         
 11.39.06 STC15284  AB5C4- REGS  0 - 1  = 00000000_00002000   00000000_84306000                                                       
 11.39.06 STC15284  AB5C4- REGS  2 - 3  = 00000000_00000000   00000000_00000000                                                       
 11.39.06 STC15284  AB5C4- REGS  4 - 5  = 00000000_00000000   00000000_008FFCD8                                                       
 11.39.06 STC15284  AB5C4- REGS  6 - 7  = 00000000_000000FF   00000000_00000000                                                       
 11.39.06 STC15284  AB5C4- REGS  8 - 9  = 00000000_7F0E7068   00000000_013F6D38                                                       
 11.39.06 STC15284  AB5C4- REGS  10- 11 = 00000000_00000000   00000000_00000000                                                       
 11.39.06 STC15284  AB5C4- REGS  12- 13 = 00000000_00000000   00000000_7F0E7068                                                       
 11.39.06 STC15284  AB5C4- REGS  14- 15 = 00000000_84306000   00000000_0000000C                                                       
 11.39.06 STC15284  AB5AA-  S306 ABEND AT 000BA2 IN CSVABEND, IL=2, R15=0000000C                                                      
 11.39.06 STC15284  AB5AA- SNAP5A ID: LP=3D7FF598, LEVEL=05/16/2023, 07.09, MXA, WA=3D6D2EB0.                                         
 11.39.06 STC15284  AB5C5- LAST LOADED ABEND-AID MODULE WAS #XAAHZTB                                                                  
 11.39.06 STC15284  AB5C7 DUPLICATE SVCDUMP SUPPRESSED BY DAE                                                                         
 11.39.06 STC15284  AB5C8 SYMPTOM STRING = PIDS/ABENDAID LVLS/         RIDS/*UNKNOWN                                                  
 11.39.06 STC15284  AB5C8   ADRS/000BA2 AB/S306 PTFS/        LVLS/           LVLS/                                                    
 11.39.11 STC15284  CCSR010E XCOMSTSK S306 at 00000000 LMOD N/A CSECT N/A +N/A XCOM N/A SERVER                                        
 11.39.11 STC15284  CCSR061I PSW: 00000000 00000000 070C1000 813F6812                                                                 
 11.39.11 STC15284  CCSR062I ILC: 02 INTERRUPT CODE: 0D                                                                               
 11.39.11 STC15284  CCSR067I COMPLETION CODE S306 REASON CODE 0000000C                                                                
 11.39.11 STC15284  CCSR064I DATA AT PSW 013F680C : 8400181E 0A0D18FB 180C181D                                                        
 11.39.11 STC15284  CCSR065I HOME = 00B8  PRIMARY = 00B8  SECONDARY = 00B8                                                            
 11.39.11 STC15284  CCSR070I GR00 - GR01  00000000_00002000 00000000_84306000                                                         
 11.39.11 STC15284  CCSR070I GR02 - GR03  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR04 - GR05  00000000_00000000 00000000_008FFCD8                                                         
 11.39.11 STC15284  CCSR070I GR06 - GR07  00000000_000000FF 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR08 - GR09  00000000_7F0E7068 00000000_013F6D38                                                         
 11.39.11 STC15284  CCSR070I GR10 - GR11  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR12 - GR13  00000000_00000000 00000000_7F0E7068                                                         
 11.39.11 STC15284  CCSR070I GR14 - GR15  00000000_84306000 00000000_0000000C                                                         
 11.39.11 STC15284  CCSR071I AR00 - AR03  00000000 00000000 00000000 00000000                                                         
 11.39.11 STC15284  CCSR071I AR04 - AR07  00000000 00000000 00000000 00000000                                                         
 11.39.11 STC15284  CCSR071I AR08 - AR11  00000000 00000000 00000000 00000000                                                         
 11.39.11 STC15284  CCSR071I AR12 - AR15  00000000 00000000 00000000 00000000                                                         
 11.39.11 STC15284  CCSR066I SDWAEC2 00000000 00000000 from RB                                                                        
 11.39.11 STC15284  CCSR062I ILC: 00 INTERRUPT CODE: 00                                                                               
 11.39.11 STC15284  CCSR070I GR00 - GR01  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR02 - GR03  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR04 - GR05  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR06 - GR07  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR08 - GR09  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR10 - GR11  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR12 - GR13  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  CCSR070I GR14 - GR15  00000000_00000000 00000000_00000000                                                         
 11.39.11 STC15284  XCOMM0053E  ưQ     ABEND S306  REASON 0000000C IN MODULE(N/A     ), OFFSET(+013F6810)       

In order to remediate the issue, the XCOM started task needed to be recycled and everything went back to normal.

SYSOUT and CEEDUMP provided.

Resolution

The SYSOUT shows a S0C4 abend related to the SSL handshake client hello i.e.
*****
 CEE3204S The system detected a protection exception (System Completion Code=0C4).                                                    
          From entry point format_client_hello at statement 4937 at compile unit offset +00000274 at entry offset                     
          +00000274 at address 3CC5BF9C.                                                                                              
*****

XCOM Engineering advised:

1. The root cause of the problem is an invalid timestamp being received on the SSL handshake from the XCOM partner. Somewhere within the XCOM transfers it is likely there is an old XCOM for Linux system running an old OpenSSL version which is passing the invalid timestamp on the SSL handshake.
   When we have seen problems with these timestamps in the past, they have been very intermittent. The same XCOM partner that sends an invalid timestamp also sends valid ones 99% of the time. Unfortunately, the CEEDUMP does not show enough storage to be able to determine the IP address of the remote partner.
   The CEEDUMP shows that the abend occurred in the IBM C "doformat" function which was called by the IBM C "strftime" function.  This abend is not in XCOM code and the best he can do is try to catch the invalid timestamp that is coming from the Linux/UNIX partner system.
   
   Traceback:                                                                                                       
  DSA   Entry       E  Offset  Statement   Load Mod             Program Unit                   Service  Status   
  4     doformat    +000019EA              CEEEV003                                            HLE77D0  Exception
  5     __strftime_std                                                                                           
                           +000000E6              CEEEV003                                            HLE77D0  Call     
  6     strftime      +00000082              CEEEV003                                            HLE77D0  Call     
   
   A support case should be opened with IBM for the S0C4 abend in the "strftime" function.
   IBM will require the CEEDUMP and they may also need a SYSMDUMP. 
   NOTE: SYSMDUMPs must be written to a standard DATASET and must not be sent to SYSOUT.
   With a valid SYSMDUMP XCOM Engineering could potentially use that to track down what is incorrect about the timestamp in the buffer contents and then be able to change the XCOM code to handle the timestamp as a workaround before it gets into the IBM code. However ideally IBM should fix their code to prevent the S0C4 abend.
   
   
   
2. From all appearances, the reason the address space becomes unresponsive is due to the recursive S306 abends caused by Abend-AID attempting to load one of its modules from a library configuration that it is expecting to be APF authorized. Disabling it should eliminate those cascading abends, and thus eliminate what seems to be the cause of the hanging server.
   This DD statement will disable Abend-AID in a particular address space and if used in the XCOM JCL will disable Abend-AID only in those XCOM jobs if the S0C4 abend reoccurs:
   //ABNLIGNR DD DUMMY