The PSLwriter is crashing and in the logs, an error indicates that it is unable to allocate memory.
Business Service Insight 8.3
First, determine if this is Oracle which is not able to allocate memory or .NET on the APP server. If the error contains anything that starts with 'ORA', this means it is probably an error coming from the Oracle database. Anything about allocating tablespace means one of the Oracle tablespaces is out of memory and needs to be expanded. That needs to be addressed on the Oracle side.
A possible exception might be something like:
INTERNAL ERROR (ORA-0) in open...
That out of memory error might be coming from the Oracle client saying it did not have enough RAM to store any query results from the database.
If you see this or if the error comes from the PSLwriter and is not an Oracle database error, then .NET or the Oracle client ran out of RAM on the APP server.
The simplest resolution to this problem in the event that the PSLwriter ran out of RAM is to reduce the max event block size in the advanced settings. This determines how many events the PSLwriter will try to save to RAM at one time:
For example, reduce this to 500 or in some rare cases even lower; the lower limit is 1. Note that lowering this value may reduce performance, as it then requires the PSL Writer engine to load and unload events from the database more frequently. It does, however, cause the engine to use less RAM. The aim is to ensure that the PSL Writer is not going much above around 1GB of memory usage, which you can see in the Task Manager. If you are using a lot less, you can increase the event block size and process a greater number of days per cycle for better performance.
Now, while that will resolve the issue it is also important to discuss other things that affect performance and RAM usage. These include:
1. The number of PSL instances on the system. In general, we recommend one (1) instance per CPU thread and 2GB RAM.
2. The amount of actual RAM on the server. We never suggest running a production APP server with less than 8GB RAM. Also, check what other applications are installed and using RAM on the system. Ideally, an APP server should be a dedicated APP server.
3. The size of the events. Obviously, 1000 events which contain a handful of integer fields are going to take much less RAM than 1000 events which contain large strings. Events should contain only the data being used for calculations. You should avoid large descriptions or string fields which are not needed.
4. The engine settings are shown here:
There are several settings here which can also affect RAM.
1. The number of consecutive days setting, which you can see is 100 in the above screenshot. Reducing this can reduce the number of events being loaded. However, if you have a large number of events then lowering this value may still not reduce the blocks of 1000 events you are loading into the system and therefore this setting by itself may not help solve this problem.
2. The upper and lower memory bounds for the engine. It is important to understand that these are not hard limits. When the engine loads events to calculate, it is going to use as much RAM as it needs. This may be MUCH more than the upper bound. However, when it finishes the calculation if it finds itself above the upper bound then it will attempt to free memory until it is below the lower bound. So these may not affect an out of memory issue, but if they are not set at all then we would recommend setting them if you get these errors. You may start at 2000 and 1000 and lower them further if needed, but this really depends a lot on available RAM and other factors.
In the event of an ORA-0 out of memory error coming from the Oracle client, then the above settings may help but this may also be a more general problem on the machine (since this is coming from the Oracle client and not the PSLwriter). In that case, check task manager and make sure there is not anything left open like sqlDeveloper which is using all the RAM on the system.