Error Message :
Symbolic links pointing the Applications Manager Agents output directory are generated for every PeopleSoft job that is run and remain there even after the job has finished. The links will be cleared only after stopping and restarting the Agent where the PeopleSoft jobs ran.
When looking at the number of open file descriptors in the /proc/fd directory, you will see the number of open file descriptors continuously growing. If you look at one of the files you will see that the referenced link points to an output file in the $AW_HOME/out directory on the Agent where the PeopleSoft jobs are running.
Fri Oct 18 09:03:28 PST 2013
>ls -l 559
l-wx------ 1 saprd psoft 64 Oct 15 09:19 559 -> /home/AM8 /out/o31636297.00
>ls -l o31636297.00
-rw-r--r-- 1 saprd psoft 405 Oct 15 00:18 o31636297.00
When looking at the file itself, you may see that the output file is from a job that has finished days earlier.
Start time: 10/15/13 12:17 AM
Run ID: 31636297
Tue Oct 15 00:17:03 EDT 2013
Requested using UF_PROCSCHED
Run Control ID:UF_SF_SI_LEG_OUR_INCR
Tue Oct 15 00:17:03 EDT 2013 Process scheduled with instance:1692493
Tue Oct 15 00:17:03 EDT 2013: Job: 1692493 status:Queued(31)
Tue Oct 15 00:18:13 EDT 2013:Job final status: Success(32)
In PeopleSoft we process the output separately from the job completion. If the job does not have an output scan and output is not required (*1) then as soon as the job completes the job can drop to history and we retrieve and register the output in the background and update it to the job later. In that case, we leave the output file open until that post-processing of the output completes.
What we found is that there was a change in 2011 for an OAE issue which affected the PeopleSoft job processing (*2). The change affected the logic so that the system output file was getting closed only if there was an error retrieving the output. In other words, it was not getting closed in most normal cases. In addition, due to this issue, the job never got removed from the status monitor queue which would affect memory and performance.
*1. Output required is the ‘Abort if no output’ checkbox on the job definition.
*2. Applications Manager v8.0 SP7: #182698: Fixed a problem where OAE tasks got stuck in a RUNNING status when the connection to the RmiServer process dropped during task completion.