Two Linux servers were recently upgraded to SPE 126.96.36.199
Shortly thereafter there were lots of container violation errors and failed scans (as per the SPE logs)
In early hours of the morning, usually between 3am and 6am there would be memory spikes reported by a 3rd party monitoring software.
The spikes only lasted 12-15 minutes, then subsided. It did not occur every day. The spikes reported almost always occurred on the primary server.
Release : 188.8.131.52
Based on activity in the logs there were a combination of factors contributing.
1a: There were thousands of Scan Errors-- mostly container violation errors of the following type:
- Antivirus 11 - Failed with scan error hex code 0x80800204
Antivirus 11 usually means that the SPE engine was given a file to scan but it was unable to access the file to scan it, or that it no longer existed in the location (especially when files are transient). In this case they were hosted by a NetApp device where the files submitted were snapshots and didn't really exist as a file. The user account to the NetApp device was also locked or unable to provide access to the files in the location they were supposed to be.
1b: Encrypted container
The files were either really encrypted or, as per the Antivirus 11 condition inaccessible to the point that they were determined to be encrypted. Usually the files were really encrypted/password protected which makes them inaccessible.
2. Memory allocated to the servers (both VMs) was 16GB.
The combination of errors en masse and base memory allocated caused the server to be overworked due files being repeatedly resubmitted for scanning. This caused the symcscan process to consume more and more memory until it recycled and started the process over.
Cause 1a: Access to the location of the files on the repository eliminated the problem with the same files being resubmitted repeatedly which prevented the servers from being overworked.
Cause 1b: Encrypted files do not generally cause a problem other than, if there are a lot of them, they can create more work for the scanning engine. There is a configuration that can be made via the Cloud Console to tell the Scan Engine to not bother with encrypted files.
- Log into the Cloud Console (assuming the server is enrolled to be managed by the Cloud Console)
- Locate the Policy that is applied to the Group to which the server is assigned and click "Edit" at the top.
- Select the section "Archive Handling"
- Uncheck the box "Enable encrypted file archive handling" and "save"
Cause 2. The server's allocated memory was adjusted to 32GB