TCP Tunnel keeps disconnecting for users accessing PAM application with ZTNA

Products

• Symantec ZTNA

Issue/Introduction

ZTNA setup to access a number of TCP applications using TCP tunnels. 

All tunnels seem to work fine, except for one - Symantec Privileged Access Manager (PAM).

TCP tunnel setup on ZTNA Portal for PAM application, with both target and local TCP port set to 443. When the SSH tunnel is initially established, all TCP traffic destined to the PAM endpoint will be routed through the local loopback address to the final destination. After a few minutes however, a message appears on the tick PAM application indicating that the connection to server was lost. 

This disconnect seems to be happening for all users on Windows (a handful of users on macOS do not appear to report issue).

Netstat output on Windows host confirms huge numbers (1000s) of TCP connections to the 127.0.0.1 TCP 443 SSH application in either the TIME_WAIT or CLOSE_WAIT state, eventually causing resource exhaustion and disconnects.

Environment

ZTNA.

Secure Access CLoud.

Symantec Privileged Access Manager client application.

TCP Tunnel to back end PAM application.

Cause

Interoperability issue between PAM and ZTNA.

Symantec ZTNA TCP tunnels and the CA PAM both use loopback address 127.0.0.1. When ZTNA tunnel established, the internal flow of the PAM client is negatively impacted resulting in it continuously closing and and trying to establish new connections. Some of these connections are not closed in an elegant manner and remain in TIME_WAIT or CLOSE_WAIT status for as long as the Windows timeout permits. By default  timeout value for improperly disconnected TCP/IP connections is two (2) hours. 

In MacOS the OS reacts to idle connections by gracefully closing the communication in both ends and hence the problem is not observed.

Resolution

Use ZTNA RDP Application to connect to an internal Windows jump server host running PAM, and access sites from there.

Engineering is working on an improved integration between PAM and ZTNA.

Additional Information

Another workaround possibility is to change the value of the timeout period for Windows, using the following registry modification:

• “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters” key

• Locate the Value “KeepAliveTime”. If it does not exist, create a new REG_DWORD Value called “KeepAliveTime”

• Set the Value “KeepAliveTime” to decimal 120000

NOTE that this will impact all TCP connections to any given server and hence it may not be desirable. Although it will certainly decrease the number of open connections, it will not eliminate the problem completely