Looping Messages stuck in O365 Queue after DLP Cloud Service Migration to GCP

Products

• Data Loss Prevention Cloud Service for Email

Issue/Introduction

Using the DLP Cloud Service for Email, with O365 in Reflecting mode, you are seeing errors after the service was migrated to Google Cloud Platform.

Messages are queueing up in O365, with the error "421 4.3.0 Loop Detected" - and the queue is building rapidly and not clearing.

Full error example:

Reason: [{LED=421 4.3.0 Loop Detected. Check reflect mode configuration: https://support.symantec.com/en_US/article.DOC9008.html.};{MSG=};{FQDN=smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com};{IP=144.49.246.194};{LRT=1/27/2021 5:13:57 PM}]. OutboundProxyTargetIP: 144.49.246.194. OutboundProxyTargetHostName: smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com

This continues to occur, even after making the required change* for the IP range of  144.49.240.0/21 in the Transport rule for sending messages through the DLP "Outbound Connector" in O365.

 

*As per the Broadcom Product Advisory for the GCP Migration

Cause

Root cause of the looping is because you forgot to include the new IP address range in the list of exceptions in your Transport Rule - this causes messages already handled by the DLP Cloud Service to be resent by O365 back to DLP, which then resends it back to O365 via your Inbound Connector.

However, the reason the queue is not clearing is due to Microsoft's Transport "logic": Once messages are first assigned by a Transport Rule to a Connector, the messages will no longer receive updates to the Transport ruleset.
If they have been re-queued by O365 and in a "Pending" status, the messages will still be following the initial ruleset, and will thus continue to be sent with the outdated rule.

Environment

Release : 15.x

Component : DLP Cloud Service for Email; only customers using O365 Reflecting mode are impacted by this issue.

Resolution

The Transport rule does need to be updated, as per the Broadcom Product Advisory for the GCP Migration, and some time may be required for this change to take effect (up to 30 minutes, according to Microsoft documentation)

NEW messages sent through the Service with the updated Transport rule will correctly utilize the newly added IP range and be sent without errors.

Message which were already queued, owing to "Loop detected" error condition however must be sent via a different Connector configuration - one that skips the DLP Cloud Service, sending straight to intended recipients.

Because the already-queued messages are assigned a specificly named Connector, the Microsoft advice to address this is as follows:

1. Create a copy of the Outbound DLP Connector which sends messages to DLP - saving it with a different name.
2. Update the existing DLP Transport rule to use the new Connector in step 1. Ensure it has been updated with the IP Exceptions noted in the advisory above.
3. Edit the original Outbound DLP Connector so that it no longer sends messages to the DLP Cloud Service for Email. In most cases this change would be from "a" to "b" below:
   1. Deselect "Route mail through these smart hosts."
   2. Change the routing instead to whichever setting will send mail out to the next hop after O365 - in most cases, this will be set to use MX records.

The goal here is to route queued messages to skip DLP Cloud Service (because of the Microsoft Transport issue), and simultaneously to update the existing Transport rule to use a new Connector, so that NEW messages will be sent to the DLP Cloud Service as expected.