Error getting connection to user store. User store up and running.

Products

• CA Identity Manager
• CA Identity Governance
• CA Identity Portal
• CA Identity Suite

Issue/Introduction

3 IM servers. one of the servers (node 2).  We found the below error in server.log but all 3 user store servers are up and running. The errors are gone after restarting the jboss service on node 2. 

Error getting connection to user store." while processing policy with member rule "<MemberRule><AttributeExpression attribute="%ADMIN_ROLE_CONSTRAINT%" comparator="EQUALS" value="Provisioning Manager for JDBC-WBDDRussiaConfig"/></MemberRule>". Continuing with NO MATCH assigned for this policy.
2020-11-11 12:32:14,344 ERROR [ims.llsdk.directory.jndi] (http-156.81.251.252:8080-27) Search failed with filter (&(uid=80196683)(objectclass=inetorgperson))
2020-11-11 12:32:14,344 ERROR [ims.llsdk.role.azcache] (http-156.81.251.252:8080-61) Exception "[facility=4 severity=3 reason=0 status=6 message=Unrecognized command]
Error getting connection to user store." while processing policy with member rule "<MemberRule><AttributeExpression attribute="%USER_ID%" comparator="EQUALS" value="80146427"/></MemberRule>". Continuing with NO MATCH assigned for this policy.
2020-11-11 12:32:14,379 ERROR [<Hostname>] (http-156.81.251.252:8080-50) <Hostname>.ADGroupMgmtUtility(Method:getGroupOwner) - inputfilter:: LDAP$$IDX$$USER$$ou=people,o=<Company name>,dc=pep,dc=pvt$$(&(sn=*Gupta*)(|(pepUserType=EMPLOYEE) (pepUserType=INTL EMPLOYEE)))$$uid&&givenName&&sn&&mail&&pepUserType&&pepDivision&&pepsubdivision&&pepLocName&&pepUserStatus&&pepRegistered
2020-11-11 12:32:14,389 WARN  [ims.tasktrack.custom] (Worker_actionq#ActionQ1#WPDS_2018) AR_59027379 - MyidmARFWFParticipantResolverEngine getAppApproverTypes() apprTypesStr: Approver1:Manager/Security Assistant,Approver2:GROUP OWNER APPROVAL,Approver3:WINDOWS DOMAIN ADMINISTRATOR,Approver4:AD GROUP OWNER APPROVAL,Approver5:PADR PLATFORM OWNER,Approver6:ISUM ACTIVE DIRECTORY,Approver7:WINDOWS DOMAIN ADMINISTRATOR,Approver8:AD ENTERPRISE ADMINISTRATOR
2020-11-11 12:32:14,402 WARN  [ims.default] (Thread-14346 (HornetQ-client-global-threads-1659285896)) No email template could be found for event "SetPrimaryObjectAuditEvent", email will not be sent
2020-11-11 12:32:14,415 ERROR [ims.llsdk.role.azcache] (Thread-14362 (HornetQ-client-global-threads-1659285896)) Exception "[facility=4 severity=3 reason=0 status=6 message=Unrecognized command]
Error getting connection to user store." while processing policy with member rule "<MemberRule><AttributeExpression attribute="%USER_ID%" comparator="EQUALS" value="09163745"/></MemberRule>". Continuing with NO MATCH assigned for this policy.
2020-11-11 12:32:14,420 ERROR [ims.llsdk.role.az.search] (http-156.81.251.252:8080-39) _dirDoesManagedObjectMatch getting search on subject "cn=Montage Virtual Interview,ou=Applications,ou=groups,o=<Company name>,dc=pep,dc=pvt" throws error: [facility=4 severity=3 reason=0 status=6 message=Unrecognized command]
Error getting connection to user store.
2020-11-11 12:32:14,425 ERROR [ims.llsdk.role.azcache] (http-156.81.251.252:8080-58) [facility=4 severity=3 reason=0 status=6 message=Unrecognized command]
Error getting connection to user store.
 at com.netegrity.llsdk6.imsimpl.directory.jndi.JndiSearcher.<init>(JndiSearcher.java:120)
 at com.netegrity.llsdk6.imsimpl.directory.jndi.JNDIDirectoryProvider.getSearcher(JNDIDirectoryProvider.java:808)
 at com.netegrity.llsdk6.imsimpl.securityengine.AuthorizationCacheManager._evaluateDirectoryPolicies(AuthorizationCacheManager.java:680)
 at com.netegrity.llsdk6.imsimpl.securityengine.AuthorizationCacheManager.calculatePolicies(AuthorizationCacheManager.java:1165)
 at com.netegrity.llsdk6.imsimpl.securityengine.AuthorizationCacheManager._calculatePolicies(AuthorizationCacheManager.java:642)
 at com.netegrity.llsdk6.imsimpl.securityengine.AuthorizationCacheManager.evaluatePolicies(AuthorizationCacheManager.java:394)

Environment

Release : 14.2

Component : IdentityMinder(Identity Manager)

Resolution

User stores and IM continued to be up and running. The root cause was a sporadic and possibly one-off issue somewhere in the network or firewalls. Please review the date\timestamp with your network team further.  Whatever happened the network caused the loss of connection between the IM and working user store. Restart the Application server allowed this connection to work again.