All jobs remain in Launch wait - error: " o_ord_cycle | Last IO operation failed with error 3. End of cycle"

Products

• CA Automic Dollar Universe

Issue/Introduction

Error Message :
On the universe.log these kind of errors may appear all the time

###########################################
| 2014-08-05 11:50:56 |ERROR|X|IO |pid=1252.5348| o_ord_cycle | Last IO operation failed with error 3. End of cycle
| 2014-08-05 11:50:56 |ERROR|X|IO |pid=1252.5348| loopDetection | Loop detection for engine tid: 624 code: L, 1091 loops in the same minute
###########################################

Patch level detected:Dollar Universe 6.2.00
Product Version: Dollar.Universe 6
At some point, the launcher stops submitting new jobs and all jobs remain in Launch Wait Status.
It can occur when there is a power outage and the server hosting Dollar Universe is suddenly shutdown.

Environment

OS: All

Cause

Cause type:
Other
Root Cause: The errors on the universe.log are related to some corrupted information on any of the launcher data files:
u_fmsb60/ u_fmsp60 / u_fmtp60

As a result, the launcher can not submit any more jobs.

Resolution

In order to allow Dollar Universe to submit jobs agains, please reinitialize the launcher data files via the following procedure.

Solution:
1. Copy / Paste the jobs in status "Launch Wait / Event Wait" from UVC - Job Runs or via the command "uxlst fla full" to know what was supposed to be launched next.
2. Shutdown the Dollar Universe and check that all process are down
3. Backup the launcher data files of the concerned area (u_fmsb60.dta, u_fmtp60.dta, u_fmsp60.dta)
4. Load the Dollar Universe environment
5. Reinitialize the launcher data files of the concerned area by executing the following commands under bin

Warnings
~~~~~~~~~~
-THE IMPACT OF THIS MANIPULATION WILL BE THE DELETION OF THE PREVIOUS LAUNCHES.
-THE LAUNCHES GENERATED BY A SCHEDULED TASK MUST BE RECREATED AFTER THE ENGINES ARE STARTED.
-THE MANUAL LAUNCHES OR THE LAUNCHES GENERATED BY AN UXORDRE WILL HAVE TO BE MANUALLY RECREATED.

Example for the area EXP (X):

uxrazfic u_fmsp60 X
uxrazfic u_fmsb60 X
uxrazfic u_fmtp60 X

6. Reorganize the data files and then start the node again, launching the following commands from bin
unireorg
unistart

7. In case the task with "Start Date" in the past are not recalculated, launch the following two commands:
uxupd tsk tsk=*
uxupd tsk tsk=* ses=*
 

Fix Status: No Fix

Additional Information

Workaround :
N/A