active directory account auth is failing. users are unable to connect.

Products

• CA Harvest Software Change Manager

Issue/Introduction

harvest was not upgraded last night. best we can tell from our change ticket system neither did AD. This morning no one can connect. The only account that works the is the harvest account. 

tried some diagnostics with hauthsync....

[email protected]:/opt/CA/scm/log $ hauthsync -b prdscm01 -usr harvest -pw xxxx 
[email protected]:/opt/CA/scm/log $ cat hauthsync.log
I00060040: New connection with Broker prdscm01  established.
E0202011d: Authentication operation failed: Referral .
E03060048: External Authentication synchronization failed.
[email protected]:/opt/CA/scm/log $

hauthtst isn't working either

Environment

Release : 13.0.3

Component : CA HARVEST SCM INFRASTRUCTURE (BROKER/AGENT/PEC/SECURITY

Resolution

Active directory administrators had introduced a change.  

Reviewing his LDAP directory structure, his basedn was "DC=erieinsurance,DC=com"

At this level there are 3 "referral" links:
- ldap://********.com/CN=Configuration,DC=********,DC=com (this one is working)
- ldap://DomainDnsZones.********.com/DC=DomainDnsZones,DC=********,DC=com (this one is not working)
- ldap://ForestDnsZones.********.com/DC=ForestDnsZones,DC=********,DC=com (this one is not working)

As a test, we changed the "-ldapbasedn" to a lower level node that included most of the Harvest users but did not include the above "referral" links.  Hauthtst was able to succeed in authenticating users when we did this so we made the same change in HServer.arg and bounced the broker.  Now the majority of users are able to login.  

Active directory administrators backed out their change to provide the permanent fix.  HServer.arg was restored to its original configuration and the broker restarted again.  Now all users can login.