Multiple Gen EJB Web Services in single WebSphere ear file delayed first response

Products

Gen

Issue/Introduction

Using Gen 8.6 EJB Web Services under WebSphere (Traditional) Application Server (WAS) 9.x:

When assembling a single EJB Web Service into a single ear file for WebSphere, then deploy and test the web service from SOAP UI, the first request/call responds successfully in a timely manner.
When assembling multiple (approximately 20) EJB Web Services into a single ear file for WebSphere, then deploy and test one of those web services from SOAP UI, the first request/call seems to wait forever/hang and eventually times out with " java.net.SocketTimeoutException". If the timeout setting in the SoapUI request is set to a large value e.g. 3600000 milliseconds (1 hour), the response will eventually come back in 20+ minutes. Subsequent requests also have much quicker responses.

NOTE: The problem does not occur when deploying to JBoss EAP.

Environment

Gen 8.6 Enterprise Java Beans under WebSphere Traditional Application Server

Resolution

Per: MustGather: JAX-WS/JAX-RPC Web Services engine problems for WebSphere Application Server
Under WebSphere 9.x Administration Console, for the server being used enabled Diagnostic trace with higher Log Details level i.e.
*=info: com.ibm.ws.webservices.*=all: org.apache.*=all: com.ibm.ws.websvcs.*=all: com.ibm.ws.metadata.*=all"
After restarting WebSphere and executing the request from SoapUI, the generated trace.log showed exception:
+++
org.apache.axis2.jaxws.message.databinding.JAXBUtils JAXBContext_newInstance Exception thrown from AccessController: java.security.PrivilegedActionException: javax.xml.bind.JAXBException: Provider com.ibm.xml.xlxp2.jaxb.JAXBContextFactory could not be instantiated: javax.xml.bind.JAXBException: error loading class "GqabPersonalInfoSvrcall" listed in ejbpilot/jaxws/jaxb.index, make sure that entries are accessable on CLASSPATH and of the form "ClassName" or "OuterClass.InnerClass", not "ClassName.class" or "fully.qualified.ClassName"
+++

The trace shows that it goes on to use a list of classes to build the JAXBContext and eventually the Fallback JAXBContext is created:
+++
[10/12/20 23:47:43:453 PDT] 0000010b JAXBUtils 1 org.apache.axis2.jaxws.message.databinding.JAXBUtils createJAXBContextValue Building a JAXBContext with packages only success=false
[10/12/20 23:47:43:453 PDT] 0000010b JAXBUtils 1 org.apache.axis2.jaxws.message.databinding.JAXBUtils createJAXBContextValue Try building a JAXBContext using a list of classes.
[10/12/20 23:47:43:454 PDT] 0000010b JAXBUtils 1 org.apache.axis2.jaxws.message.databinding.JAXBUtils createJAXBContextValue Start finding classes
...
...
[10/12/20 23:47:56:073 PDT] 0000010b JAXBUtils 1 org.apache.axis2.jaxws.message.databinding.JAXBUtils createJAXBContextValue Successfully created JAXBContext Primary JAXBContext:
bundleresource://137.fwk491475464/com/ibm/xml/xlxp2/jaxb/JAXBContextImpl.class,
Version: 1.6.6-jaxb,
Timestamp: Tue, 13 Mar 2018 14:56:19 EDT,
Classes known to this context:
[NONE] (Fallback JAXBContext will be used to process any requests.)

Fallback JAXBContext:
bundleresource://11.fwk491475464/com/ibm/jtc/jax/xml/bind/v2/runtime/JAXBContextImpl.class Build-Id: null
Classes known to this context:
...
+++

The time to build the JAXBContext explains the excessive time being taken before the response is received in SoapUI.

The use of a jaxb.index file was first created in a Gen 8.0 PTF BTN80018\RO55723 (WEB SERVICES PERFORMANCE ON WEBSPHERE) to aid with the speed of building the application JAXBContext for earlier versions of WebSphere.
This problem will be investigated further by Gen Engineering team, with a view to providing a new fix.
Until then having a single EJB Web Service per ear file is the recommended approach.

Additional Information

This behaviour has been found to be due to a change between Gen 8.5 & 8.6.
During the 8.5 Build Tool Assemble step there is additional code generated in the model directory jaxws (located under directory java\package_name) which is related to annotations for the model generated code. The annotation code is pure java code and does not depend on the Gen runtimes and is also compiled and placed inside the server manager .jar files inside the assembled .ear file.
That additional annotation code is no longer generated and included during the 8.6 Build Tool Assemble step.
As a workaround these steps have been found to be successful if requiring to have multiple EJB Web Services in a single ear file:

Generate, build and assemble the 8.5 model for the required EJB Web Services.
Generate, build and assemble the same 8.6 model code. Take a backup of the .ear file that is created for later comparison.
Under the 8.5 model directory copy the jaxws directory from directory java\package_name and copy it to the same directory under the 8.6 model directory.
Run the 8.6 assemble step again. The resulting ear file will contain server manager jar files that contain a jaxws sub-directory with the jaxb.index file and annotation classes.
Compare to a server manager jar file from #2 which has also has a jaxws sub-directory but only has the jaxb.index file.
Install the new .ear file into WebSphere and the hang/timeout problems should be resolved.

A PTF is planned to be released for Gen 8.6 but due to other prioritised committed work, there is no timeline on when this might be available.

In addition for improved WebSphere JAXB class caching performance, these 4 JVM custom properties are advised to be set:
com.ibm.ws.websvcs.getJAXBContext.cacheClassList = true
com.ibm.ws.websvcs.getJAXBContext.cacheClassList.persist = true
jaxws.JAXBContext.permStoreCostThresholdMsec = 5000
jaxws.JAXBContext.permStoreMaxSize = 1000
They can be set Using the WebSphere Admin Console under:
Servers -> WebSphere application servers -> SERVER_NAME -> Java and Process Management -> Process definition -> Java Virtual Machine -> Custom properties

January/February 2026 UPDATE

In another reported scenario with a ear file containing 50 EJBs the above parameters enabled a first response reduction from 9 minutes to 2 minutes.

The problem is that WebSphere is scanning all class files in the ear file, so reducing the number of class files in the ear file could potentially provide some further improvement i.e. try to assemble the ear files with fewer EJBs.
The user subsequently found that using these 2 custom JVM properties work better than the above 4 custom JVM properties:
com.ibm.jtc.jax.xml.bind.v2.bytecode.ClassTailor.noOptimize=true
com.ibm.jtc.jax.xml.bind.v2.runtime.JAXBContextImpl.fastBoot=true
Just using the 2 properties gives less than 1 minute for the first EJB call with an average of 35 seconds.
The properties are documented in IBM article JAXBContext Initialization Takes A Long Time (viewing requires an IBM account login).
The article covers the disabling of pre-caching and optimization using the properties and it advises:
*****
Here the rule-of-thumb for determining an appropriate pre-caching feature for an application is based on the re-usability of the JAXB context(s):
- If the re-use factor is high, enable pre-caching (a slow boot, but boosted runtime performance).

- For heterogenous and non-repeatable JAXB operations, disable pre-caching to gain boot time performance.
*****
Gen Engineering advised they would expect the applications to still execute successfully using those properties.
Also the user advised that the performance of subsequent calls was not impacted