Log and report on large number of emails using Data Protection

Products

• Email Security.cloud

Issue/Introduction

This article offers a solution for basic logging and reporting of all emails that pass through the Email Security .cloud Service (ESS), when there are large numbers involved.

The scope of the solution is as follows:

• The customer account has the Data Protection service provisioned
  • The solution uses this service for logging all emails
• The Track and Trace tool cannot fulfil the expected task, due to the inherent limitations of this tool
  • This tool is designed for specific searches through the live databases of our email processing infrastructure. It cannot do large scope searches and provide consistent reports using wide ranging criteria that would yield results in the thousands
• Delivery logs are not a mandatory part of the required task
  • The Data Protection service cannot log and capture information regarding the post-processing delivery state of emails, as it processes email data during its transit through the ESS, before delivery is even attempted

The solution consists of configuring a Data Protection policy that will log all inbound or outbound emails (or both) starting from the moment it is activated. Data logged by this policy is fully indexed and held on the service infrastructure for up to 40 days. You may therefore use the Email Track and Trace tool, using the Service filter for Data Protection, to quickly and consistently find and list emails that have triggered the policy up to approximately 1000 results. Past this number, you may use the Reports feature of the portal to obtain reports on emails passed through the service, with up to 500.000 lines of results per report.

Note: This articles predates the ETDR product offering which includes Data Feeds. As such, if you have ETDR, and by extension access to Data Feeds, via it you can get all of the information as well directly into your SIEM, email delivery data is also available. Access to Data Feeds renders this policy unnecessary.

Resolution

Data Protection policy

Steps to configure the policy:

• Access Services > Data Protection
• Create a New Policy
  • Give it a descriptive Name: Email Traffic Logging
  • Apply to: Either inbound OR outbound email | Suggestion to split the policies, one for inbound and one for outbound 
  • Execute if: ALL rules are met
  • Action: Log Only
  • Click Edit next to the Notification option, check the box Use custom notification and disable all notifications, click Edit
• Add a new Rule
  • Execute if: ANY conditions are met
• Add a new condition - Match all

OPTIONAL: By default, the Data Protection service logs the following information against the emails it triggers for: Date, Envelope Sender Address, Envelope Recipient Address (only one of the recipient addresses, if more are present) and Subject. If you require more detailed information (HELO, IP, Message-ID, Reply-to and Body-From) to be logged and stored from the headers of the emails that the policy triggers against, proceed with the next step of the guide. Otherwise, you may save and activate the policy as it is and move on to the Report configuration section.

• Add a new condition - Content Regular Expression List
  • Click on Create a new Regular Expression List
  • Name the list: Capture Header Information
  • Add the following lines to the list using copy/paste into the Add list items section, click Add and then click Save

(?:1)?\(((?:HE|EH)LO(?!=)\n?(?:.*?))\)[\s]{1,2}?\((?:\d{1,3}\.){3}\d{1,3}\).*?\n?.+?messagelabs\.com
(?:2)?\(((?:\d{1,3}\.){3}\d{1,3})\).*?\n?.+?messagelabs\.com
(?:3)?^(From:(?:.*?\n){1,4}?)(?=^[\w-]+?:)
(?:4)?^(Reply-to:(?:.*?\n){1,2}?)(?=^[\w-]+?:)
(?:5)?^(Message-ID:(?:.*?\n){1,3}?)(?=^[\w-]+?:)

• Condition options:
  • Email contains: a number of matches for the regexes in the selected lists
  • At least: 1
  • Count only unique matches: No
  • Case sensitive: No
  • Look in: Header
  • Matched text: Log Matched text
• Change the Execute if value for the rule from " ANY conditions are met" to " All conditions are met"
• Click on Save
• Find the policy in the list (automatically created at the end) and click on Activate next to it

NOTE: You may consider changing the position of this policy in the Data Protection service policy list. The policy will only log the emails that have reached it, after they will have been processed by all other policies before it, some of which may be configured to Stop evaluation of lower priority policies.

Reporting

OPTIONAL: If you configured the extra reporting condition Content Regular Expression List in the Data Protection policy, there are certain steps that you need to take in order to ensure that the report will also contain that data. NOTE: For the customer accounts that are provisioned under a partner, or you’re a partner creating these policies/reports on behalf of your client, your partner account won’t work to create this report. You need to use a local user of the account with the correctly configured role.

1. Ensure that you are using the main user of the account (example form ABC1234) or a secondary user with the role (permission) to View Sensitive statistics
2. Ensure the following options within the Data Protection configuration page are activated
   • Access Services > Data Protection
   • Click on Settings
   • At the bottom of the Settings page, in the Reporting section, please select both options (Show matched content on reports and Show surrounding text on reports)
   • Click on Save

Steps to configure the report:

• Access Reports > Report Requests
• Click on Request a new report
• Give it a descriptive Name: Email Traffic Reporting
• Select Email Detailed Report (CSV) - Data Protection
• Click on Advanced Settings
  • Type or copy/paste the exact name of the Data Protection policy that you created before under Policy Name (example: Email Traffic Logging)
  • Click on Add
• Click on Continue and configure the time interval for which the report will gather the data; you may also Schedule the report to run automatically
• Click on Continue and configure the delivery method for the report
• Click on Continue, review the configuration of the report and click on Submit request