Increasing HB.js zIIP workers and impact to the memory usage

Products

• HB.js

Issue/Introduction

How to increase the number of HB.js zIIP worker threads and what is the impact on memory usage of increasing this parameter?

The current setting is 16 which matches the parameter TASKMAX=16.

Resolution

1. The number of zIIP tasks can be raised or lowered by changing parameters in the HBZ@BASE or HBZ@JSE parmlib members, depending on which pool of threads need altering (HBz PARMLIB Members).
Setting the TASKMIN and TASKMAX parameters will control how many tasks are allocated at region start and how many it can grow to as additional threads are needed. 
When running off the zIIP, memory for HB.js services is allocated via CICS and comes from EUDSA.  When running on the zIIP, memory for HB.js services is allocated via z/OS and comes from the total memory pool for the CICS job.  So when running on the zIIP, memory is allocated in the CICS STC/JOB but outside of CICS's view/control.

The number of threads that can be run is dependent on a couple of factors.
 - How much storage is being used, on average, by the HB.js zIIP threads
 - How much storage is available to be allocated by HB.js zIIP threads.
This is storage that is not allocated by CICS but is available in CICS STC/JOB i.e. storage allocated to the STC/JOB but not used by CICS.

2. Engineering reviewed the resulting storage report from running commands HBZT STOR,0 and HBZT STOR,3.
Looking at the report they believe the maximum number of zIIP tasks can be increased by at least 8 and probably 16.
Increase it by 2 for a few days before increasing it by another 2. At the end of each day, examine the number of zIIP threads in use. If it did not reach the maximum number of zIIP threads for a day then stop increasing the parameter.
Looking at the data they believe that 16 threads would be sufficient most days, but only if no threads go offline.


ADDITIONAL:
zIIP tasks use storage inside the CICS started task but outside of CICS's control which means if the zIIP threads run out of storage the CICS region may ABEND and not issue an SOS (the current release issues a script "out of memory" runtime message and will not ABEND), so care must be taken when increasing the number of tasks. Each task will take at a minimum 4MB of storage. The total amount of storage consumed by a thread is determined by the script the task is running. Generally most scripts will run in the 4MB without needing any additional storage, but depending on the script it can consume 100MB or more. Understanding how much storage scripts may consume is critical before increasing the number of zIIP tasks.
Using SysView the storage can be seen as it is allocated and released by the zIIP threads. It is recommended to manually profile script storage usage to determine a worse case storage usage scenario.The HB.js zIIP tasks allocate storage in two subpools, the initial allocation of 4MB is in user subpool 1,subsequent allocations are in user subpool 2. Allocations are in extended private space and key 8.

To view storage allocations using SysView

• Use the ASID command to select the target CICS region:
• Use the Private command to view the specific subpool

ASID HBCICSD               --  Target region
PRIVATE ALLOC,ALL,USR,1    --  User subpool 1 - Static 4MB per zIIP task
PRIVATE ALLOC,ALL,USR,2    --  User subpool 2 - Dynamic allocations

Other non-HB.js tasks will also allocate storage in these subpools. Profile the maximum storage usage by any zIIP tasks in subpool 2. Once you have what you believe to be the maximum usage by any script, use that number to calculate a "worst case scenario" for storage usage.  Take the maximum storage used in subpool 2 by a zIIP task, add 4MB (the storage allocated in subpool 1) and multiply that by the number of zIIP threads being started. That will give the amount of storage required if all zIIP threads are running the largest memory consuming script.  

How much memory is available for scripts?  

Job/started task region size minus EDSALIM - sit parameter or it's default value minus 100MB for other CICS overhead equals available script storage.

Example:

Job/STC region = 1024MB

EDSALIM = 800MB (CICS TS 5.6 default)

1024 - 800 - 100 = 124MB of storage for scripts

Additional Information

Example JCL for running a sample SYSVIEW batch job:

//SYSVIEW  EXEC PGM=GSVXBAT,PARM='LINECNT=66,SHOWINP=NO,MENUOFF' 
//SYSPRINT DD  SYSOUT=*                                          
//SYSIN    DD *,SYMBOLS=JCLONLY,DLM=$$                           
COMMAND=(ASID HBCICSD)                                           
COMMAND=(PRIVATE ALLOC,ALL,USR,1)                                
COMMAND=(PRIVATE ALLOC,ALL,USR,2)                                
$$                                                               
//*