The below messages were generated from MIMGR, due to System SYSA being unable to Access VCF.
09:28:48.23 MIM0061W System SYSC in file VCF - possibly inoperative
09:29:47.23 MIM0200W VCF ACCESS DELAYED - MASTER=SYSA
09:29:47.23 MIM0328W VCF Reserve held by: SYSC
09:29:51.67 MIM0061W System SYSB in file VCF - possibly inoperative
09:29:51.67 MIM0061W System SYSD in file VCF - possibly inoperative
Customer does not see a cause for the contention.
This has been occurring for over an hour at this point.
A GRS,C display on SYSA shows
RESPONSE=SYSA
ISG343I 13.59.24 GRS STATUS 322
S=SYSTEM ARCENQG ARCMCDS
SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS
SYSA MIMGR 0024 007B5168 EXCLUSIVE OWN
SYSA DFHSM 0064 008FED90 EXCLUSIVE WAIT
NO REQUESTS PENDING FOR ISGLOCK STRUCTURE
NO LATCH CONTENTION EXISTS
This indicates that MIMGR has issued a blocking ENQ on SYSA indicating the ARCENQG QNAME is in use on another system.
This causes DFHSM on SYSA to wait for the resource. This is proper behavior on the part of MIM.
Issuing this multiple times did not show any change.
MIMGR and MIA are UP in Status and show no other errors.
As above HSM is in WAIT Status, similar contention is being seen on the respective other LPARs in the SYSPLEX
Looking for assistance on the outstanding reserve request for the VCF
MII
There can be many causes for VCF delays.
In this specific instance, the messages and problem persisted for a longer period of time with no visible cause, no additional errors on the side of MIM, and consistent results in the GRS,C displays for the DFHSM QNAME/RNAME of ARCENQG ARCMCDS showing contention.
Because the contention does not appear to change, issuing the following commands on all systems can help determine exactly which system owns the DFHSM RNAME.
GRS,C
/F MIM D CONFLICTS
/F MIM,D ENQR=(ARCENQG ,ARCMCDS)
Issuing the commands multiple times can also determine whether any of the ENQs are changing on any of the systems indicating whether the problem may be resolving itself, or whether there is the potential for a deadly embrace.
Review of the syslog where the MIM0328W is occurring (in this case SYSC), and going back to before the first time the message occurred may indicate other problems that are causing delays for DFHSM (or any other task that may appear to be hung).
In this case, the SYSC syslog was showing abends and problems in several other major tasks such as CICS and DB2.
Once those problems were resolved, the repeated displays noted above showed the problem with DFHSM resolving itself until there was no longer any contention.