Enable crash dumps on Windows Server 2019 for hub.exe and controller.exe crashes

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

Microsoft Windows error/event ID 1001 hub and controller crashing every 3-4 hours in DX UIM 23.4 CU2.

Fault bucket, type 0

Event Name: APPCRASH

Response: Not available

Cab Id: 0

Environment

• DX UIM 23.4 CU2

Resolution

If the hub and controller are not running 23.4 CU2 yet, please make sure you have upgraded the hub and robot (controller) probe to 23.4.2 first from 23.4.0 and follow all of the steps listed below. 

Check whether crash dumps are enabled or not on the hub server, and if not, use the instructions to enable crash dumps so that a memory dump of the hub.exe process will be collected automatically if the hub unexpectedly restarts.

1. Open "regedit" on the server in question and navigate to the following registry location:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

2. If the "LocalDumps" folder does not exist in the registry, create it. 

3. Now, inside this folder you should create the following registry keys/values for Name, Type and Data.

i.    "DumpFolder" as a REG_EXPAND_SZ to something like C:\Dumps\ 

ii.   "DumpCount" as a REG_DWORD to 10 // this will limit to a maximum of 10 dump files

iii.  "DumpType" as a REG_DWORD to 2 // this creates a full process dump 

 

4. Reboot the hub server for the registry changes to take effect.

5. Set the loglevel for the hub and controller to 5 using Raw configure and add/set the logsize key to 30000; do not change any settings on the hub after this.

6. Deactivate the Nimsoft Robot Watcher Service and take a backup of the hub and robot folders under the Nimsoft folder.

    The debug build versions of hub and controller have been attached to this Article but this process for collecting crash dumps can be used for other non-debug-enabled hub/controller probes.

       hub-23.4.2-debug.zip

       controller-23.4.2-debug.zip

7. Unzip and extract all of the files in the zips and copy all of the files contained in each .zip to the nimsoft/hub and nimsoft/robot folder. 
      
8. Copy controller.cfg from nimsoft/robot to nimsoft/robot/changes folder and then remove the "magic_key" full line as shown below from the hub section in nimsoft/robot/changes/controller.cfg.

<hub>
  description = Message concentrator and redistributor
  group = Infrastructure
  active = yes
  type = daemon
  command = hub.exe
  config = hub.cfg
  logfile = hub.log
  workdir = hub
  magic_key = vnE7EmoDI3pRE0lDHx7oN83b29xSjoEKCXpELeiRQLPtPIflWs2o4f8/+YsjYNKcC6UOqwwXy5J/e75/TPvxlciI8RdIiyi2sfatLRDHmOMeY9oAWzpZ4ZLFBPwqdUGWYZhRz7gV4924tXlTdOAJ1cTvgGXn0j50jomeuEIg75+Fb+OMvgGjsuhZn0Dmpj8o2QUJ7xIe5KJqGW9mJGP5yg==
</hub> 

9. Set active = 0 under <plugin_metric> under hub section in hub.cfg file as below.
   <plugins>
     <plugin_metric>
        pluginPath = plugins/plugin_metric/plugin_metric.dll
        pluginName = plugin_metric
        configPath = plugins/plugin_metric/plugin_metric.cfg
        configSection = variables
        active = 0
     </plugin_metric>
   </plugins>        
  
10. Start the Nimsoft Robot Watcher Service and when a hub/controller crash occurs, immediately collect the files listed below:

• hub.log
• _hub.log
• controller.log
• _controller.log

In the folder specified earlier, e.g., C:\Dumps\, dump files should have been created. 

Please zip up and attach all the files from this folder as well as the hub/controller logs and attach them to the support case.