Secondary SpectroSERVER crashing after an online backup sync with the Primary SpectroSERVER

Products

• Spectrum
• Network Observability

Issue/Introduction

Secondary SpectroSERVER crashing after an online backup sync with the Primary SpectroSERVER. 

The following is seen in the WinDbg stack trace of the SpectroSERVER.mdmp file:

STACK_TEXT:  
00000037`aaeff2a8 00007ffe`085e701d     : 00007ffe`085fd160 00000037`aaeff3e0 00007ffe`0860369a 00007ffe`0862da40 : msvcr120!MoveSmall+0x152
00000037`aaeff2b0 00007ffe`085f3716     : 00007ffe`0862da40 00000000`0000000f 00007ffe`085fd160 00000000`0000000b : mtsfm64!_msize__+0x3dd
00000037`aaeff2e0 00007ffe`085f33d5     : ffffffff`00001300 00007ffe`0862da40 00000000`00000000 deadbeef`deadbeef : mtsfm64!mts_bad_free+0x6a6
00000037`aaeff380 00007ffe`085e435e     : 00007ffe`1f9863b4 00000037`aaeff418 00000101`ffffffff 00007ffe`085fd160 : mtsfm64!mts_bad_free+0x365
00000037`aaeff3c0 00007ffe`085e6cec     : 00007ffe`1bb3e010 00007ffe`1f9863b4 00000000`00000000 00007ffe`1f9548c7 : mtsfm64!mts_enable+0x183e
00000037`aaeff400 00007ffe`0ee4dcec     : 00000037`aaeff4a0 00000037`aaeff570 0000018f`e2943ec0 00000037`aaeff570 : mtsfm64!_msize__+0xac

Environment

Version: Any version running on Windows
Component: SpectroSERVER

Cause

There are two probable causes for this crash:

• By default, Spectrum running on Windows uses the SpectroSERVER_mts version of the SpectroSERVER binary. This version of the SpectroSERVER uses a third party memory manager.
• The anti virus software running on the system is hitting the files in the Spectrum root ($SPECROOT) directory.

Resolution

Implemented the SpectroSERVER_nomts binary

• Shutdown the SpectroSERVER process
• Copy the $SPECROOT/SS/support/SpectroSERVER_nomts file to the $SPECROOT/SS directory
• Rename $SPECROOT/SS/SpectroSERVER_nomts to $SPECROOT/SS/SpectroSERVER
• Restart the SpectroSERVER process

 

Excluded $SPECROOT from anti virus scans