How to address an Error when soap query returns more than 2mb of data.
Article ID: 44226
Updated On:
Products
Issue/Introduction
When trying to fetch data from a web service whenever the size of the response is very large, the following error is seen:
Reason :Operator Aborted: Response for this Operator exceeds threshold. A new value can be defined (in MB) by adding 'oasis.response.payload.size.threshold' in OasisConfig.properties file. Having larger threshold can result in memory issues and is NOT recommended.".
Environment
Component:
Cause
This is due to a performance limitation within Process Automation. A Dataset value cannot be larger than 2mb in size. When this is increased beyond 2mb, the memory load on the Orchestrator increases and we very quickly run into performance issues.
For versions of ITPAM 4.3 SP04 CP01, the limit has been increased to 4mb.
As an example, from internal testing, when we attempt to read a 6mb file into a dataset we could no longer open the instance run to review the dataset, and this is a minor issue. If you make heavy use of dataset values larger than 2mb, this could quickly increase the load on the orchestrator to the point where it cannot complete any instances.
Resolution
To help handle larger payloads of data, there are a couple of suggestions:
1. Modify your soap call to return less data; for example instead of pulling in all users, you could query for all users from A-M in one query, and the N-Z users in a second query and act on each separately.
2. Do this work outside of the Orchestrator; for example, you could use Powershell to query the web service, gather and format the data and return a more limited amount of data into PAM. Or the Powershell script could do all the work and just report the results to the pam process.
Additional Information
There is a payload setting that can be used to increase the amount of data, we do not recommend modifying this as it could produce extremely negative performance consequences, including causing the orchestrator to completely stop responding altogether.
This is provided strictly for informational purposes: oasis.response.payload.size.threshold and the value would be a numeric as 6 - for instance
oasis.response.payload.size.threshold=6
This is not a recommendation to make this change, the setting is provided as informational only.
Feedback
