Tunnel fails to connect with message: CTRL send message failed for new conn / get id commands
Article ID: 211366
Updated On: 03-05-2024
Products
Issue/Introduction
A newly installed tunnel client is not connecting to the tunnel server. On the client hub, the following message is seen in the hub.log:
Feb 18 11:10:37:582 [140353004885760] 0 hub: CTRL send message failed for get id command
Feb 18 11:10:37:582 [140353004885760] 0 hub: CTRL failed to get ID from ##.##.##.##/48003
or:
Feb 21 14:59:07:304 [140558869669632] 0 hub: CTRL send message failed for new conn command (ip:port)
Feb 21 14:59:07:304 [140558869669632] 2 hub: CTRL hubname new (inner) connection failed
Environment
Release : 20.3
Component : UIM - HUB
Resolution
This error ultimately means that the tunnel connection is failing either during or shortly after the initial SSL handshake.
We have observed a number of causes for this behavior:
1. MTU Mismatch between hub client and server
2. Existing IPSEC Tunnel connection or "site-to-site VPN"
3. Deep Packet Inspection
4. Corrupt tunnel certificate on client
Each of these issues is described in more detail below.
MTU MIsmatch:
If the MTU settings on the client do not match those on the server, this error message can occur.
The specific MTU value may not be important - the default for many systems is 1500 but we have seen successful connections with MTU set to 1460 or 1300; but it should match on both endpoints (tunnel client and tunnel server)
Example:
Using the "ifconfig" command on Linux we find that the MTU on the tunnel client has been set to 8900 (also known as "jumbo frames" size):
[root@myserver] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 8900
The following command implements the change to 1500:
[root@myserver]# ifconfig eth0 mtu 1500
after running this command we can run "ifconfig" again to verify the result:
[root@myserver]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
Existing IPSEC Tunnel or site-to-site VPN
Several customers have attempted in the past to set up a UIM hub-to-hub tunnel across an existing site-to-site VPN connection, or a "tunnel within a tunnel" without success.
For various reasons related to the routing of packets on such networks, UIM tunnels do not work well across this type of connection, and the error message above is often seen when the tunnels try to connect.
If an existing site-to-site route exists in your network environment, we recommend using Static Hub Routes instead of tunnels, which will not rely on the additional SSL layer that often causes problems with tunnel connectivity.
Deep Packet Inspection
In some environments, additional IP Security measures may be implemented, such as SSL Decryption or Deep Packet Inspection, which are implemented at the router level to provide an extra layer of scrutiny and security for packets traversing the network/firewall.
These measures can interfere with reliable and timely packet delivery and may cause timeouts and latency with the SSL handshake which can lead to the given error message appearing in the logs during the tunnel handshake. Generally speaking, UIM traffic (e.g. port 48003) should be placed on a device's "allow list" to pass traffic without such additional inspection or decryption being attempted.
Corrupt tunnel certificate on client
In most cases, when a tunnel certificate is corrupt or invalid, an error message specific to the certificate decryption will be seen, but in rare cases we have encountered a behavior where the error messages above appear in the logs, but the first three causes listed here are not applicable. In such cases, deleting the tunnel client configuration and certificate, re-issuing the certificate at the tunnel server, and re-building the connection from scratch resolved the issue without further intervention at the network level.
Feedback
