Tunnel Failure after upgrading hub to 23.4.7 / CU7

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

After upgrading a tunnel server or client hub to 23.4.7 from any prior version, the following symptoms may occur:

• tunnel server not starting - hub not listening on tunnel port (e.g. 48003)
• tunnel clients not connecting to tunnel server
• telnet test to tunnel port 48003 fails
• error in hub.log:  
  
  ssl_log_error - Could not read public key file., [1] error:0x0A00018E: SSL routines: func(167772558): ca md too weak
  
  
• alarm received:
  
  "Failed to start NimBUS tunnel server"

Environment

• DX UIM 23.4.7
• hub 23.4.7
• SSL tunnels configured
• Tunnels were set up on hubs prior to 23.4.0

 

Cause

OpenSSL upgrade mandates SHA384 signature algorithm for hub certificates 

Resolution

A hotfix (hub 23.4.7.1) for this issue is available at the CU7 Solution Document page.

 

If you have a backup of the system or the /hub/ folder:

If you have taken a backup of the system, or the /Nimsoft/ or /hub/ folders prior to the upgrade, take the following steps:

1. Restore the backup to downgrade the hub back to the prior version
2. Download hub 23.4.7.1 into the Archive, but do not deploy it.
3. Edit the hub.cfg on every tunnel hub and in the <tunnel> section, add the key:  enable_cert_compatibility = yes
4. Deploy hub 23.4.7.1 and it will allow the SHA1 certificates to work.
5. Once everything is back up and running, follow the steps below to reset the tunnel server CA and re-issue the client certificates.

Note: you may also connect locally to the afffected hubs using Infrastructure Manager, and downgrade them to hub 23.4.6 or prior, rather than restoring a backup.

If you do not have a backup, or you are unable to downgrade, or if you have completed the above steps and now need to replace the certificates with SHA384:

In order to resolve this issue, the tunnel server CA will need to be reset, and all existing tunnel client certificates will have to be invalidated, and new certificates issued to replace them.

If you have upgraded the Tunnel Server only, but no clients:

If you have upgraded your tunnel server hub to 23.4.7 but the clients are still on 23.4.6 or earlier, you can temporarily downgrade the hub back to a prior version (23.4.6 or earlier).  This will allow the tunnel clients to reconnect.

Then, before upgrading any tunnel hub to 23.4.7 you will need to recreate the Tunnel Server CA and issue new client certificates.

Now you must distribute the new certificates to the clients.

If you happen to have tunnel redundancy (two tunnel servers per client), you can use one tunnel to upgrade the certificates for the other.  This process is described in further detail here.

Additional details on using a superpackage to distribute new certificates is available here.

 

If you have upgraded one or more Tunnel Clients, but not the tunnel server:

If you have upgraded one or more tunnel clients to 23.4.7 and they have gone offline, it will be a little more difficult to bring them back.

You will need to take the following steps:

1. Install a copy of Infrastructure Manager locally on the client, or in the case of a Linux client, on a Windows machine in the same local network.
2. Point the IM Client to the upgraded client hub and log in using administrator credentials.
3. Import an earlier hub version (23.4.6 or prior) to the Archive on the client hub, and then distribute it to the hub using drag-and-drop in Infrastructure Manager.

Once the hub is downgraded to 23.4.6 or earlier, it will come back online using the existing client certificate.

Once all the clients are back online, you will need to go through the process of recreating the Tunnel Server CA and re-issuing the client certificates as described in the section above this one.