Spectrum database corruption after a system reboot

Products

• Spectrum

Issue/Introduction

By default, when processd is shutdown during a system reboot, processd will attempt to shutdown the SpectroSERVER and Archive Manager processes gracefully before it shuts down. However, out of the box, processd will only wait 20 seconds for the SpectroSERVER and Archive Manager to shutdown before it shuts itself down. If the SpectroSERVER or Archive Manager processes take longer than 20 seconds to shutdown, the OS may kill these processes resulting in database corruption.

This knowledge article discusses how to configure processd to wait longer than 20 seconds to ensure the SpectroSERVER and Archive Manager processes are shutdown gracefully when the system is rebooted.

Environment

Any Spectrum version running on any supported OS

Resolution

Windows:

Set the WaitToKillServiceTimeout parameter in the Windows registry to a higher number. By default it is set to 20 seconds.

The following is from the "SpectroSERVER Shutdown in a Windows Environment" section of the documentation:

In a Windows environment, processd waits until all subprocesses have shut down before it stops. However, the Windows registry setting WaitToKillServiceTimeout sets the length of time that Windows waits for all services to stop after a Windows shutdown is initiated. If a service such as processd is still running after the WaitToKillServiceTimeout elapses, Windows terminates this service. The default value is 20 seconds, which is not always enough time for the SpectroSERVER to shut down completely at system shutdown. You can increase the timeout value.

Follow these steps:

1. Open the Registry Editor.
2. Go to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control.
3. Click the Control key.
4. Double-click the WaitToKillServiceTimeout value in the right pane
5. In the window, change the value to anything up to 600,000 milliseconds (10 minutes)
6. Click OK
7. Restart the Windows server for the change to take effect.

Linux:

Add the PROCESSD_SHUTDOWN_TIMEOUT parameter to the /opt/SPECTRUM/spectrum80.env file. This parameter is not there by default so you have to add it manually. This file is owned by the root user so you will need root access to change this file.

For example, to set the wait time to 60 seconds, add the following parameter to the /opt/SPECTRUM/spectrum80.env file:

PROCESSD_SHUTDOWN_TIMEOUT=60000

After making this change, you will need to manually shutdown the SpectroSERVER and Archive Manager processes and then shutdown and restart processd.