What is causing a deadlock on ACF2 Infostorage database that resulted in a hang in other processes?

Products

• ACF2
• ACF2 - z/OS
• ACF2 - MISC

Issue/Introduction

A JESPLEX was blocked for more than a minute. The Display GRS command showed ACF2 as the top blocker which had an exclusive ENQ on the Infostorage database. What information is revealed from the GRS output, specifically the cause of this problem?

 

Environment

Release : 16.0

Component : CA ACF2 for z/OS

Cause

An SVC dump of the ACF2 STC during the hang can expose the conditions that lead to this lockout.

For example, in one svcdump it was seen that the OMVS profile directory had nearly 2.4 million OMVS profile records. Upon further investigation,  over 2 million logon IDs had just been added. At the time of the SVCdump a 

'F ACF2,REBUILD(USR),CLASS(P),DIV(OMVS)'

command was in progress. The REBUILD process had done an exclusive ENQ on the INFOSTG database so those 2+ million records could be read. The resulting I/O in situations like this is not likely to complete within just one or two minutes.

Resolution

The addition of new OMVS PROFILE does not have to be followed by a REBUILD command.  ACF2 is designed so that OMVS profile record administration is able to dynamically determine the OMVS profile fields without the need for doing a REBUILD for OMVS user profiles.

 

Additional Information

There are two main functions that require the OMVS profile fields. One is the initUSP service which creates the USP (user security packet) for USS. The other function is the RACROUTE EXTRACT call which can request OMVS profile information. Both functions are designed to work correctly after adding OMVS profile records even if a REBUILD has not been issued. 

ACF2 handles initUSP calls by checking a UID table extension for newly added UIDs. The original UID table is built when ACF2 initializes and is sorted by UID. When an administrator adds new OMVS profiles the UID values are added to the UID extension table when the INSERT command is issued, so there is no need to do a REBUILD. ACF2 checks the primary UID table (sorted) and then the UID table extension when it needs to determine a user's UID for initUSP events. 

For RACROUTE EXTRACT ACF2 will first check to see if the OMVS profile information for a given user is in the directory structure. If not, the EXTRACT processing will check to see if the record has been added to the INFOSTG database. This approach allows EXTRACT to handle newly added OMVS profile records without the need for a REBUILD.