Tunnel Failure after upgrading hub to 23.4.7 / CU7

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

After upgrading a tunnel server or client hub to 23.4.7 from any prior version, the following symptoms may occur:

• tunnel server not starting - hub not listening on tunnel port (e.g. 48003)
• tunnel clients not connecting to tunnel server
• telnet test to tunnel port 48003 fails
• error in hub.log:  
  
  ssl_log_error - Could not read public key file., [1] error:0x0A00018E: SSL routines: func(167772558): ca md too weak
  
  
• alarm received:
  
  "Failed to start NimBUS tunnel server"

Environment

• DX UIM 23.4.7
• hub 23.4.7
• SSL tunnels configured
• Tunnels were set up on hubs prior to 23.4.0

 

Cause

OpenSSL upgrade mandates SHA384 signature algorithm for hub certificates 

Resolution

In order to resolve this issue, the tunnel server CA will need to be reset, and all existing tunnel client certificates will have to be invalidated, and new certificates issued to replace them.

If you have upgraded the Tunnel Server only, but no clients:

If you have upgraded your tunnel server hub to 23.4.7 but the clients are still on 23.4.6 or earlier, you can temporarily downgrade the hub back to a prior version (23.4.6 or earlier).  This will allow the tunnel clients to reconnect.

Then, before upgrading any tunnel hub to 23.4.7 you will need to recreate the Tunnel Server CA and issue new client certificates.

If you have a small number of clients, you can choose to manually distribute the new client certificates. However, this is much easier to do if you can set up a second tunnel server, especially if there are a large number of clients.

The high level process is as follows:

1.  Set up a second tunnel server on hub 23.4.6 or earlier
2.  Connect each tunnel client to this new tunnel server -- now each tunnel client will have two active tunnels.
3.  Now you can reset/renew the CA Certificate on the first tunnel server, which will invalidate all the client certificates and cause the initial tunnels to the first server to fail, but the second tunnel will still be active.
4.  At first the clients may appear red or disappear from IM, but after about 1-2 hours the tunnel routes should stabilize and you should be able to contact the clients again via the second tunnel route.
5.  Once this happens, you can reset the Server CA, issue new client certificates from the first tunnel server, and then reconfigure the clients by replacing the certificate in the first tunnel connection, because you will still be able to configure the remote hubs using IM across the secondary tunnel.

This process is described in further detail here.

Additional details on using a superpackage to distribute new certificates is available here.

 

If you have upgraded one or more Tunnel Clients, but not the tunnel server:

If you have upgraded one or more tunnel clients to 23.4.7 and they have gone offline, it will be a little more difficult to bring them back.

You will need to take the following steps:

1. Install a copy of Infrastructure Manager locally on the client, or in the case of a Linux client, on a Windows machine in the same local network.
2. Point the IM Client to the upgraded client hub and log in using administrator credentials.
3. Import an earlier hub version (23.4.6 or prior) to the Archive on the client hub, and then distribute it to the hub using drag-and-drop in Infrastructure Manager.

Once the hub is downgraded to 23.4.6 or earlier, it will come back online using the existing client certificate.

Once all the clients are back online, you will need to go through the process of recreating the Tunnel Server CA and re-issuing the client certificates as described in the section above this one.