Pam site is inaccessible but the cluster appears to still be in sync

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

At some point in time end users stopped being able to login into the primary cluster site. The servers where pingable and port 443 remained open on all primary site nodes. When trying to connect to these nodes directly the login fails. All problematic PAM appliances were in the same ESX cluster but not running on the same host. The cluster status on the remaining nodes showed a green status. The 

Testing access to the PAM Appliances from within the same subnet found extreme delays in communication. Using a simple openssl connect command we were able to determine that communication was possible but extremely slow.

[root@server ~]# openssl s_client -connect 10.10.10.10:443
CONNECTED(00000003)

About 2 min minutes later we could see the connection was finally completed

 

[root@server ~]# openssl s_client -connect 10.10.10.10:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = US, O = CA, CN = xceedium.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, O = CA, CN = xceedium.com
verify return:1
---
Certificate chain
 0 s:C = US, O = CA, CN = xceedium.com
   i:C = US, O = CA, CN = xceedium.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBozCCAUigAwIBAgIJAIJuxxxxxxxxxxxxxxxxxxxxiNl2n7U
-----END CERTIFICATE-----
subject=C = US, O = CA, CN = xceedium.com

issuer=C = US, O = CA, CN = xceedium.com

 

After additional testing with different machines inside and outside this subnet we found communications like RDP and SSH were capable when connecting from the outside to the inside but machines inside the network subnet could not initiate the same outside.

Cause

After much testing we found several physical network ports were recently enabled on one specific network switch. This ultimately caused a network routing issue causing delayed communication for only certain packets. Since there was no direct error with the system, only delays, there was no specific reason to suspect this switch.

Resolution

Removing the unnecessary ports in the network switch restored normal network communication allowing for access to PAM services.