SMSMSE 7.10 - SAVFMSESp - crashing

Products

• Mail Security for Microsoft Exchange

Issue/Introduction

After the upgrade on prod. Environment SMSMSE to v.7.10 - SAVFMSESp.exe crashing and messages are queued.

The warning messages on exchange servers - Event1050: The execution time of agent 'SMSMSERoutingAgent' exceeded 493511 milliseconds while handling event 'OnSubmittedMessage' for the message with InternetMessageId: 'Not Available'. This is an unusual amount of time for an agent to process a single event. However, Transport will continue processing this message.

Action 19:49:42: fnSetSMSERegistryAndFileSecurity. 
Action start 19:49:42: fnSetSMSERegistryAndFileSecurity.
-- SAVFMSEInst.dll MSI_Logging -- fnSetSMSERegistryAndFileSecurity
-- SAVFMSEInst.dll MSI_Logging -- fnCreateSecurityGroups
Action 19:49:42: . 
: Creating SMSMSE security group(s), please wait ...
-- SAVFMSEInst.dll MSI_Logging -- Failed to get remote install user value
-- SAVFMSEInst.dll MSI_Logging -- Failed function call SetSMSEFileSecurity for Cmaf folder
-- SAVFMSEInst.dll MSI_Logging -- Error creating security groups

There were errors in reading and writing to the pipes as well. In the debug view logs I could see errors 0xc0090081 (NAVE_ERR_OPENING_PIPE), 0xc0090085 (NAVE_ERR_WRITING_TO_PIPE), 0xC0090086 (NAVE_ERR_READING_FROM_PIPE). These errors could be related to permission issues.

Secondly, the force Action was being received because a command to stop savfmsesp.exe, failed with access denied (0x5) and the error condition (NAVE_ERR_PROCESS_STOP_FAILURE) was raised.  This packet was handled by sending a force scan request to newly spawned savfmsesp.exe which did not have information about the shared memory. Hence, the crash.

 

Environment

Release : 7.10

Component : Default-Sym

Cause

The issue occurs when there is a problem with the remote installation, and setting the appropriate permissions has failed 

The error related to event 378, which is related to permission 

The error related to SPA settings might indicate a permission issue on the hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\SMSMSE\7.10\Server\Components\SMTP

Because of that, you are unable to set the value of "ResetSPASpamDispositionSettings" to 0. 

You should definitely check if the permission evaluation was done before upgrading SMSMSE.

 

Resolution

1. Run the cleanup script Manual removal script for Symantec Mail Security for Exchange (SMSMSE) for all versions  

  Note: This script removes exactly the items listed in the SMSMSE Manual Removal articles.

2. Run the manual (interactive) installation.

If this fails, check the smsmse_exemonadE2K10_E2K15.log in %temp% to confirm the failure.

Check the permissions on the Folder, Registry, RBAC user, and the DCOM.

The steps are similar to the KB article - Error: "Failed to set SMSMSE registry or file security" appears during installation of Mail Security for Microsoft Exchange

Install logs

Interactive install - "C:\SmsmseSetupLogs\smsmse_verbose_install.log"

Remote install - %temp%\smsmse_inst.log

Permissions

• User permissions for 7.10
• Installation will NOT fail if the permission settings could not be done. This is because the installation permission can be added post installation also.
• Please note, SDBZ will not capture the permissions. This has to be manually verified.

SMSMSE Groups in AD

Folder Permissions

RBAC User Permissions

Registry Permissions

DCOM Permissions

Just noting the permission on the SAVFMSESpamStatsManager DCOM server.  This can be ignored in most cases. But sometimes might be worth checking. For ex: KB like this one.

Open ‘DcomConfig.exe’ (windows + R)

Navigate to Component services -> My Computer -> DCom Config.

Check the ‘access permissions’ and ‘launch permissions’ on “SAVFMSESpamStatsManager”.

AppId = {9050D46B-FBDB-4D32-8A98-EA601E03FBAE}

 

The access permissions for this appid:

 

C:\>dcomperm.exe -aa {9050D46B-FBDB-4D32-8A98-EA601E03FBAE} list Access permission list for AppID {9050D46B-FBDB-4D32-8A98-EA601E03FBAE}: Access permitted to NSTEST3\SMSMSE Admins. Access permitted to NSTEST3\SMSMSE Viewers. Access permitted to NSTEST3\SMSMSE Quar Admins. Access permitted to NT AUTHORITY\INTERACTIVE.

 

The launch permissions for this appid:

 

C:\>dcomperm -al {9050D46B-FBDB-4D32-8A98-EA601E03FBAE} list Launch permission list for AppID {9050D46B-FBDB-4D32-8A98-EA601E03FBAE}: Access permitted to NSTEST3\SMSMSE Admins. Access permitted to NSTEST3\SMSMSE Viewers. Access permitted to NSTEST3\SMSMSE Quar Admins. Access permitted to NT AUTHORITY\INTERACTIVE. Access permitted to NT AUTHORITY\SYSTEM. Access permitted to NT AUTHORITY\NETWORK SERVICE.

Log Collection

• All links
• https://knowledge.broadcom.com/external/article/177216/how-to-obtain-debug-logs-for-symantec-ma.html
• DbgView.exe can be used. Before that, CMDDebugEnabled has to be set to 1. Refer to the link above.
• During installation, smsmse_exemonadE2K10_E2K15.log will be created when PowerShell launches to check if the RBAC user’s mailbox is available. Please collect this as well.
• Sometimes scan profiling information is also needed. For that 
• create DWORD value ScanTraceEnableBool in
• HKLM\SOFTWARE\Symantec\SMSMSE\<version>\Server. 
• 0 = disable. Non-zero = enable.

CMAF log collection

https://knowledge.broadcom.com/external/article/177248/debug-common-management-framework-cmaf.html

 

Scan Processes

 

NOTE: Microsoft recommends not using processor hyperthreading with Exchange.  See this Microsoft article for more details: Understanding Processor Configurations and Exchange Performance.  Virtual hyperthreaded processors should not be counted as processors or cores in the calculations above.  The additional virtual processors should be ignored for purposes of this calculation.

Therefore, one should only consider the physical processor cores available on the system. For example, consider the below image from a test machine’s msinfo32.exe’s output:

In the above image, the value of n = 16. Consider only the ‘core(s)’ field. eg: 1 Core(s). Logical processors should not be considered.  Similarly, in the image below, the value of n = 8. 2 Core(s) * 4 Processors = 8.

When done through the installer, this is capped to 9. (we don’t want surprises during installation. There is always a risk of exhausting system resources.)

In the installation log, you can see something similar to the following:

-- SAVFMSEInst.dll MSI_Logging -- set pProcessCount to MAXIMUM_PROCESS_COUNT

 

If the count of Logical Processors is greater than cores, then we consider this to be a case where hyper threading is being enabled. We proceed with installation assuming that processor core count. The following will come up in install log:

Hyper Threading is enabled in processor, it is not recommended for Exchange

 

The below is a case of hyper threading. There are 4 cores but 8 logical processors. 

This is a non-recommended scenario. Discouraged by Microsoft themselves. Please refer to the section hyperthreading of the MSDN documentation.

 

Hyperthreading causes capacity planning and monitoring challenges, and as a result, the expected gain in CPU overhead is likely not justified. Hyperthreading should be disabled by default for production Exchange servers and only enabled if absolutely necessary as a temporary measure to increase CPU capacity until additional hardware can be obtained.

 

Transport Agents

Helpful to know the SMTP verbs. When the message hits the SMTP server, and the sending server declares that the data has ended, by pressing <CR-LF>.<CR-LF>, then the SMTPTransportAgent comes into the picture, as it handles the OnEndOfData event triggered by Exchange.

When the email arrives in the submission queue of Exchange, then the SMTPRoutingAgent comes into the picture, as it handles the OnSubmittedMessage event.