Java OutOfMemoryError in Zowe Started Task

Products

• Zowe

Issue/Introduction

The Zowe started task (typically ZOWESVR) periodically crashes or becomes unresponsive. The joblog contains the following symptoms:

"java/lang/OutOfMemoryError", exception "Failed to create a thread: retVal -1073741830, errno 132 (0x84), errno2 0xc112001e"

 

Environment

ZOWE v3.3

Cause

This error indicates that the Zowe started task (STC) has exhausted its available thread stack storage.

ZOWE Launcher procedure has MEMLMIT set to 7G, and Modulith mode is not enabled in ZOWE.yaml.

ZWELNCH  EXEC PGM=ZWELNCH,REGION=&RGN,TIME=NOLIMIT,MEMLIMIT=7G,
            PARM='ENVAR(_CEE_ENVFILE=DD:STDENV),POSIX(ON)/&HAINST.'

With MEMLIMIT set to 7GB, the memory is being contested by multiple Zowe services (gateway, discovery, caching-service, ZSS, app server, etc). When the heap memory for individual services like the gateway and discovery is set too high (e.g., 5GB), it leaves insufficient room for the remaining services and the overhead required for thread creation. 

Resolution

Step 1- Enable Single-Service (Modulith) Deployment:
Consolidation of the API Mediation Layer (APIML) services into a single JVM significantly reduces the overall memory footprint and inter-service network overhead. Add the following to zowe.yaml:

components:
  apiml:
    enabled: true

Note:  512MB heap is often sufficient in this mode. If more is required, set both heap.init and heap.max to 1024MB ( same value for both heap.init and heap.max is recommended). 

components:
  apiml:
    heap:
      init: 1024
      max: 1024

Step 2- Remove Individual Component Heap Overrides: 
To avoid configuration conflicts, remove any specific heap memory overrides previously set for the gateway and discovery components in zowe.yaml.

Step 3- Adjust or Remove MEMLIMIT:
Remove the MEMLIMIT from the Zowe STC to allow the system to manage memory dynamically. If site policy requires a MEMLIMIT, ensure the limit is sufficiently high to meet operational needs and implement a fixed thread pool for the APIML server to cap thread creation:

zowe:
  environments:
    SERVER_TOMCAT_THREADS_MIN_SPARE: 100
    SERVER_TOMCAT_THREADS_MAX: 100

Note: 100 threads are sufficient for most production environments.

 
Step 4 - Recycle ZOWE STC after applying the above changes.