SESCSP agent goes offline after 5-10 days and the ATM software hangs

Products

• Critical System Protection Client Edition

Issue/Introduction

• You installed Symantec Embedded Security Critical System Protection (SESCSP) on an Automated Teller Machine (ATM) with 1 GB or less memory.
• After 5-10 days, you find that the SESCSP agent is no longer online. Although Windows is still operational, the ATM software hangs. Only a reboot restores normal operation.
• In the logs, you find many FileWatch Collector warnings in relation to diff operations.

WARNING: 2017-01-12 10:49:29.000 Z+0200 Filewatch Collector FWC_0023: File too big to perform diff 

Environment

Windows XP
SESCSP 6.x

Resolution

1. To reduce memory usage when diff and/or checksumming is enabled, follow the SCSP DCS Best Practics for FIM Configuration on Agents.
2. If this should fail to resolved the issue, increase the amount of RAM of the system to 2 GB.
3. If the RAM increase should fail to resolve the issue, perform the following procedure and open a case with Symantec Support after you have gathered all the data:

• Prepare the system for a complete memory dump:

1. Open Registry Editor (regedit.exe).
2. Navigate to HKLM\SYSTEM\CurrentControlSet\Control\CrashControl.
3. Double-click CrashDumpEnabled and change the value to 1 (1 = complete dump, 2 = kernel dump), then click OK.
4. Navigate to HKLM\System\CurrentControlSet\Services\i8042prt\Parameters. 
5. Right-click Parameters, select New, then DWORD Value, type CrashOnCtrlScroll and press the Enter button. Double-click CrashOnCtrlScroll, give it a value of 1 and click OK.
6. Close Registry Editor.
7. Click the Start button, right-click My Computer and select Properties. Navigate to the Advanced tab.
8. In the Performance area, click the Settings button.
9. In the Performance Options window, navigate to the Advanced tab, then click the Change button.
10. If not already selected, click the Custom size radio button, then set both Initial size (MB) and Maximum size (MB) to at least the total amount of system memory + 1 MB, by entering the correct value in each field and clicking the Set button when done. E.g. if the system has 2 GB of memory, set both fields to (2 x 1024) + 1 = 2049 MB of memory.
11. After having made these changes, restart the system.
12. Download NotMyFault and unpack this to C:\Windows.

• Generate Performance Monitor logging:

1. Download and run the Performance Monitor Wizard.
2. Click Next, Next, Next, select Advanced Configuration, click Next, Next, change the log file size to 1024 MB and tick Continue logging in next file to enable circular logging, click Next, change the number of hours if necessary, click Next, click Select All in the OS Counters area, click Next, click START to start the logging process.

• When the issue reaches its peak and the ATM software becomes frozen, perform the following:

1. In the Performance Monitor Wizard, click STOP. 7-zip the output to <case number>_perfmon.7z.
2. Open a Command Prompt (cmd.exe) window and execute the command notmyfault /accepteula /crash.
3. Following the reboot, 7-zip C:\Windows\MEMORY.DMP and upload both the Performance Monitor logging and dump to your case.

Note: if no interaction with the desktop should be possible when the issue occurs, and you cannot use NotMyFault to crash the system, you will be able to force a system crash from an attached PS/2 keyboard using a right Ctrl, Scroll Lock, Scroll Lock key combination. If only the ATM software becomes frozen, please make note of its process name, as that will facilitate memory dump analysis.