Random failure to reconnect to AD managed Protection Engine servers after adding them as managed resources to the SPE Console for Windows

Products

• Protection Engine for Cloud Services

Issue/Introduction

A few Protection Engine servers were successfully added to the SPE Console for Windows to be managed. The authentication method for the servers was Active Directory (AD).

Occasionally, when opening the console again, and the program attempted to reconnect to the servers, it would fail.  An error similar to the following was found in "C:\Program Files\Symantec\Scan Engine\RestAPI\log\SPE_REST_API.log" on each of the servers where the connection failed:

ERROR spe.authentication.security.CustomLdapAuthenticationProvider:130 - CustomLdapAuthenticationProvider - authentication failed due to invalid credentials. Exception: org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: DomainDnsZones.int.example.com:636 [Root exception is java.net.ConnectException: Connection timed out: connect]]

The problem is random. Sometimes the connection attempts work, sometimes they do not.

 

Note: The above error was also seen in SPE_REST_API.log when trying to add a server to be managed and the attempt failed.

Console Error:   Unable to retrieve data from servers: servername.example.com

Environment

Protection Engine 9.x

Cause

The issue was environmental. There was a domain controller previously decommissioned but still had a record for it being a DC in DNS. 

This condition was found when analyzing a WireShark packet trace captured during a connection failure and applying one of the following filters (depending on if LDAP was configured for port 389 or 636)

tcp.port==389

tcp.port==636

The packet trace showed four unsuccessful attempts to connect to a specific address via an LDAP port (389 or 636) only to be forced to retry four times for the duration of 14 seconds.

Resolution

After removing the DNS record for the server no longer functioning as a DC, and flushing the cache of the Protection Engine servers ( ipconfig /FLUSHDNS ) the problem resolved.