Debugging ENQ (ENQUEUE) lockout Problems With the TMC/Audit

Products

• CA 1 Flexible Storage
• CA 1 Tape Management - Copycat Utility
• CA 1 Tape Management - Add-On Options

Issue/Introduction

How do I debug ENQ (ENQUEUE) lockout problems with the TMC / Audit?

 

 

Environment

Release: R14.0
Component:  CA/1 TAPE MANAGEMENT

Resolution

If you have a task that is waiting on the TMC or Audit file and is not able to continue processing you may have an ENQ (Enqueue) lockout. To diagnose the problem it needs to be determined what task is holding the resource and why it is not being released (for example, the holding task is waiting for another resource, or the holding task is swapped out). This document is to help you determine what task holds the resource.

1. Always do a "D GRS,C" on all systems. This will show any contention for resources.
   
   1. If the display shows MIMGR as the holder of the enqueue for the TMC or Audit then issue the following command:    /F MIMGR,DISPLAY CONFLICTS - This should show who is the true holder of the enqueue.
2. Determine the devices on which the TMC and Audit are allocated.
   1. Do a LISTCAT to determine which volumes contain the TMC and Audit files.
   2. /D U,VOL=xxxxxx    -   where xxxxxx is the volume of the TMC and again for the Audit. This will give you the device address for each volume.
3. Determine which system has a reserve on the TMC (or the AUDIT file).
   1. "D U,,,nnnn,1" on all systems. One system will show it as A -R, (indicates a reserve) all others will show A-BSY-P or "A" or "O".
   2. On the system where it shows A -R, issue the command:     /D,,ALLOC,nnnn,1  - this will show what tasks are allocated to the device.
4. Do another "D GRS,C" on the system that holds the reserve and look for the TOP entry after TMSQNAME TMS-TMC (or TMSQNAME TMS-AUD).
           The TOP entry should have "OWN" under the STATUS column and all other behind it will have WAIT as their status. This TOP entry on this system
           is the one that is causing the problems. Now, find out what that job it is waiting for. It might be in the CONTENTION display waiting for another
           resource (the Audit file for example). In which case you start the entire process over again and find out why it can't get the Audit file. It might be
           that a different job on a different system has the Audit file "locked" and is waiting on something else. Then that job is the real problem.
5. If the holder of the enqueue is MIMGR than issue the command:       /F MIMGR,DISPLAY CONFLICTS to see who is the real holder on the enqueue.
6. Another command that can be helpful is D GRS,RES=(TMSQNAME,*) to show all of the tasks either holding or waiting on a TMSQNAME resource.
                               If you want to show one specific resource you can enter:       D GRS,RES=(TMSQNAME,'TMS-TMC')

 

The most common causes however, are that the job that OWNS the TMS-TMC is either HSM or CATALOG. If it is HSM, you will probably have to shutdown and restart HSM. First however, look at the TCB address of HSM that OWNS the TMC or Audit file and see if you can find that same TCB waiting for a different resource. In particular, look for it under SYSZTIOT. If it is there, then you will need to have HSM shutdown and restarted. In this case, gather the SYSLOG from this system from about 30 minutes before the problem started until HSM is cancelled and everything clears out.

 

Additional Information

If the job that OWNS the TMC or AUDIT file is CATALOG, there are a few more options.

1. /F CATALOG,END(yy) - where yy is the 2-character task id. To obtain the task id issue a F CATALOG, LIST command. If this does not clear up the problem proceed to step 2.
2. /F CATALOG,ABEND(yyyyyyyy) - where yyyyyyy is the 8-character TCBADDR as in step 1. If this does not clear up the problem, proceed to step 3.
3. /F CATALOG,ABEND(yyyyyyyy),FORCE - If this does not clear up the problems then go to step 4.
4. /F CATALOG,RESTART" which should cause the catalog address space itself to be terminated and restarted automatically.

Examples:

D U,VOL=TMCVOL
IEE457I 08.06.54 UNIT STATUS 595
UNIT TYPE   STATUS     VOLSER          VOLSTATE
8720  3390    A-R      TMSVOL          PRIV/RSDNT

D U,,ALLOC,8720,1
IEE106I 08.07.00 UNITS ALLOCATED
UNIT       JOBNAME    ASID     JOBNAME           ASID   JOBNAME   ASID
8720       *MASTER*   0000     BATCHJOB          0006   LLA       0016
8720       HSM        006F     CTS               0079

 

D GRS,C
S=SYSTEM  TMSQNAME TMS-AUD               
SYSNAME   JOBNAME     ASID     TCBADDR   EXC/SHR     STATUS   CPUA 
          BATCHJOB    0175     008E36B0 EXCLUSIVE    OWN 
CPUA      TMSXTEND    0153     008E36C8 EXCLUSIVE    WAIT 
CPUA      TMSRINIT    027E     008E36C8 EXCLUSIVE    WAIT