tunnel client disconnects/resets/unstable - tunnel server shows "prolonged exhaustion of active sessions"
Article ID: 371995
Updated On:
Products
Issue/Introduction
We have a tunnel client that will not stay connected. The tunnel keeps disconnecting and then reconnecting; the hub will be available in Infrastructure Manager for a period of time but keeps turning red over and over.
The tunnel client hub.log has connection failures like this:
hub: TSESS-P-22914 could not connect to tunnel (tunnel_server_ip) (-2)
hub: CTRL (tunnel_server_name) could not connect to server (tunnel_server_ip)/53029
The tunnel server hub.log shows the following message for this client (with the client hub name in the log message):
hub: CTRL (client_hub_name) prolonged exhaustion of active sessions (0 running)
hub: CTRL (client_hub_name) prolonged exhaustion of active sessions (2 running)
Other clients are connecting to the same tunnel server without any problem. What could be the issue?
Environment
DX UIM hub - any version
SSL tunnels connecting the hubs
Cause
The underlying reason for this problem is that the tunnel client and server have been unable to establish the appropriate number of connections (usually 3 active connections and 3 passive connections on each side.)
When a tunnel starts, it immediately creates a total of 6 sessions to handle traffic across the tunnel; by default there are 3 "active" sessions which are intended to send data to the other side of the tunnel and 3 "passive" sessions which listen for data from the other side.
If the hub is unable to establish all of the "active" connections within three minutes, or if the sessions are established and something closes them, this message will occur.
Reasons for this are generally environmental and may include (but are not limited to):
- Network bandwidth throttling preventing the creation of sessions in rapid succession
- TCP Timeout settings on a firewall
- Resource contention such as an overloaded CPU/Disk/Network interface preventing the timely allocation of new sockets
This has been observed frequently in Azure environments where the Azure-hosted tunnel client is connecting to a tunnel server hosted outside the Azure ecosystem; in that case it appears related to throttling of outbound connections and the recycling of outbound ports.
Resolution
In the case where this is happening in an internally-hosted network, consult with your network teams to ensure the appropriate firewall ports are open. There may be special considerations for TCP timeouts at the firewall level as well.
In Azure, the problem can be mitigated (but not permanently resolved) by adjusting some settings in hub.cfg on the Azure clients.
This resolution may be helpful in other environments as well.
The following raw confgure keys are relevant - these would go in the hub.cfg <tunnel> section on the client hub.
hub.cfg:
<tunnel>
session_exhaustion_timeout = xxx (default - 180)
session_pool_active = x (default - 3)
session_pool_passive = x (default - 3)
These keys may be added if they are not already present:
In Azure, setting session_pool_active and session_pool_passive to '1' and session_exhaustion_timeout to '1800' can help mitigate the issue.
With these settings, the tunnel may still generate the disconnect/reset behavior as frequently as once per hour, whereas without changing the settings the behavior may occur every 3-5 minutes.
Additional Information
Feedback
