CU7 Upgrade Stuck/Failed on Starting Server phase (Secure Bus)
search cancel

CU7 Upgrade Stuck/Failed on Starting Server phase (Secure Bus)

book

Article ID: 433474

calendar_today

Updated On:

Products

DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Upgrading to DX UIM 23.4.7 fails when Secure Bus is enabled on the primary hub.

Symptoms:

  • Primary Hub has Secure Bus (hub/robot "S" version) installed
  • CU7 installer fails or stalls at 38% on "Starting Server" phase 
  • error is seen in the controller.log on the primary hub:

 Controller: ssl_log_error - Could not read public key file., [1] error:0x0A00018E: SSL routines: func(167772558): ca md too weak

 

Environment

DX UIM 23.4.7S or 23.4.7 with Hub Tunnels

Cause

Client certificates created on hubs older than 23.4.0 are signed with the SHA1 algorithm whereas hub 23.4.0 and newer signs the certificates with SHA384.

Hub 23.4.7 will not accept SHA1 certificates.

Additionally, the Certificate Authority identification certificate (ca.pem) is also invalidated and must be re-issued.

Resolution

To resolve this issue, it is necessary to re-issue the hub certificate with SHA384 format.

If you have a backup of the previous installation (or at least the Nimsoft home folder), you can restore it and then follow the steps to recreate the certificates before re-doing the upgrade.

Otherwise, you can recover the installation in progress by following the below steps:

Windows Primary Hub:

  1. Cancel the installation in progress
  2. Obtain a copy of controller.exe from the Nimsoft/robot/ folder on any other robot that is still running 23.4.0 through 23.4.6 (does not have to be Secure Robot, normal robot will also work)
  3. rename "controller.exe" in the current Nimsoft/robot/ folder to "old_controller.exe"
  4. place the older controller.exe in the Nimsoft/robot/ folder
  5. Restart the Nimsoft Robot Watcher Service
  6. launch Infrastructure Manager on the primary hub and point it to 127.0.0.1
  7. open the hub probe GUI on the primary hub
  8. Navigate to the "Tunnels" tab, and then "Server Configuration"
  9. Uncheck the "Active" box to disable the tunnel server, then click OK and restart the hub.
  10. Once the hub has restarted, open the hub GUI again and navigate back to Tunnels/Server Configuration and re-enable the "Active" box.
  11. A warning pops up asking if you are sure you want to replace the existing CA -- click Yes
  12. Enter the certificate details (most will be prepopulated other than the password - set a new password here)
  13. Delete the wildcard certificate which is under "Issued Certificates"
  14. Click New and issue a new certificate - you must use the same password as for the old certificate that you used initially here when setting up the tunnel server/secure hub (see note below if you do not rememeber this password)
  15. click OK and restart the hub
  16. Navigate to the /Nimsoft/hub/certs folder and locate the .pem file which was just created (e.g. cert02.pem)
  17. Copy that .pem file to /Nimsoft/robot/certs/
  18. Edit the /Nimsoft/robot/robot.cfg file and update the following keys which should be pointing to the "old" certificate:
    proxy_ca_location = C:\Program Files (x86)\Nimsoft\robot\certs\(filename.pem)
       proxy_cert = C:\Program Files (x86)\Nimsoft\robot\certs\(filename.pem)
       proxy_private_key = C:\Program Files (x86)\Nimsoft\robot\certs\(filename.pem)

    Change the filename here to the filename of the newly created certificate (e.g. "cert02.pem") 

  19. Navigate to the Archive and local robot_update_secure 23.4.7 and deploy it to the primary hub
  20. Now the hub and robot should start successfully and allow you to log back in with IM.  
  21. Several probes will likely be deactivated - activate any probe which is not running
  22. Re-Run the CU7 installer and the upgrade should complete.

 

Linux Primary Hub:

On a Linux Primary hub, you will need to follow the same steps as above, with a few key differences as noted here:

  • the controller binary is just called "controller" and not "controller.exe"
  • Since you cannot run IM on a Linux hub and point to 127.0.0.1, you will need to complete steps 6-15 by pointing your IM client at a nearby Windows hub that has an existing tunnel connection to the primary hub
  • As soon as you complete step 15 you will lose the connection to the Primary hub in IM until you complete the remaining steps on the Primary itself.

 

 

If you do not remember the tunnel certificate password:

If you do not remember the original password you will need to set a new password when creating the certificate, and then you will need to obtain the encrypted string for this password and paste it into the robot.cfg.

Steps:

  1. Using Infrastructure Manager, log in to any other hub in your environment
  2. Go to Tunnels->Client Configuration and create a new client connection
  3. enter "bogus" information - you do not need to enter any real IP address or certificate, but enter the password that you used for the tunnel certificate.
  4. Click OK/Save.
  5. Now open the hub.cfg and navigate to the <tunnel>  <clients> section and locate the entry for the "bogus" client you created.
  6. Copy the encrypted string from the password field here.
  7. Paste it into robot.cfg in the proxy_private_key_password field.

 

Powered by Wolken Software