Endevor Web Services started task WSTOMSTC crash
Article ID: 423784
Updated On:
Products
Issue/Introduction
Endevor Web Services started task WSTOMSTC crashed and would like to determine the root cause from the provided job log.
Environment
Endevor®
Any supported release.
Cause
In the WSTOMSTC job log are many messages "(c.ca.endevor.eapi.STCPool) [ENDEVOR] Initializing a new STC" and then messages "EDC5133I No space left on device" start. Extract of job log below:
*****...
07:21:25.681 <http-nio-18080-exec-3> INFO (c.ca.endevor.eapi.STCPool) [ENDEVOR] Initializing a new STC 07:21:44.126 <http-nio-18080-exec-17> INFO (c.ca.endevor.eapi.STCPool) [ENDEVOR] Initializing a new STC 07:21:44.182 <http-nio-18080-exec-8> INFO (c.ca.endevor.eapi.STCPool) [ENDEVOR] Initializing a new STC 07:21:44.220 <http-nio-18080-exec-5> INFO (c.ca.endevor.eapi.STCPool) [ENDEVOR] Initializing a new STC 07:21:48.831 <http-nio-18080-exec-9> ERROR (c.c.e.j.e.EndevorAPIEndpoint) [RetrieveElement - Internal error] STC: EWS1117I Request processed by SysID TEST, STC WSEWSSTC - STC11365 07:21:48.836 <http-nio-18080-exec-13> ERROR (c.c.e.j.e.EndevorAPIEndpoint) [RetrieveElement - Internal error] STC: EWS1117I Request processed by SysID TEST, STC WSEWSSTC - STC11358 07:21:48.836 <http-nio-18080-exec-9> ERROR (c.c.e.j.e.EndevorAPIEndpoint) [RetrieveElement - Internal error] com.ca.endevor.eapi.DDNameDownloadRequest.processAPIRequest:176: Download DDName APIEXTR request gets error: IOException: EDC5133I No space left on device. 07:21:48.836 <http-nio-18080-exec-13> ERROR (c.c.e.j.e.EndevorAPIEndpoint) [RetrieveElement - Internal error] com.ca.endevor.eapi.DDNameDownloadRequest.processAPIRequest:176: Download DDName APIEXTR request gets error: IOException: EDC5133I No space left on device ...
*****
Therefore the zFS file system used by Endevor Web Services has run out of space while Web Services is storing the data received by the started tasks that run the API requests.
Resolution
File system references in the job log indicate that everything apart from the CCS Tomcat binaries (catalina.home) is located under directory /cai/CADeploy/ESCM/tpv/tomcat (catalina.base).
In this scenario the syslog also contained a corresponding "ran out of space" error for the aggregate which had the same job identifier 03968 shown in the WSTOMSTC job log: M 4020000 TEST 25356 18:21:48.83 STC03968 00000080 IOEZ00551E Aggregate HLQ1.OMVS.TEST.ENDEVOR.WEBSRV.ZFS ran out of space.
The aggregate name could also be found with the OMVS "df -kv" command which will also show Available/Total space e.g.:df -kv /cai/CADeploy/ESCM/tpv/tomcat
To resolve the problem extend the available storage on the aggregate's VSAM linear data set corresponding to the Endevor Web Services zFS file system.
Feedback
