Slow Performance when Opening and Saving MS Office files from Network Shares

Products

• Data Loss Prevention Endpoint Prevent

Issue/Introduction

When DLP Endpoint Detection is enabled for Office files such as Word, Excel or Power Point on networks with high latency, a longer than expected delay may be observed.

Environment

DLP 15.x

Cause

When the "Copy to Share" channel is enabled in the Agent configuration and the Advanced Agent features of File Restoration (FileSystem.ENABLE_FILE_RESTORATION.int=1) and VEP (FileSystem.ENABLE_VEP_FILE_ELIMINATION.int=3) are at their default settings (1 and 3 respectively), an SNP copy is made in the local endpoint agent’s temp directory when a Word or Excel document is opened from a network share.

 

• When the opened file is then saved, another SNP gets created (restoration copy in case of block response rules being used) of each tmp file that the application writes to the destination share.
• This generates 2 SNP files because of the way Microsoft has designed their applications to save the working file to a temp file at the destination, and then rename the tmp file to be the expected name of the document.
• Also, a VEP copy of the tmp file that is written to the share is made in our local endpoint agent temp directory for the DLP agent to scan out-of-band to avoid sharing violations.

In this process, a total of 2 additional file copies from the share happen at Open, 1 SNP file of the actual file you are opening and 1 VEP file of the Word "Owner File" which is the hidden ~$<stub>.docx file that gets created in the same directory as the file itself, then 3 additional files copied (SNP, VEP, SNP) during Save given our default settings.

• The process (exe) seen making these SNP and VEP copies is the one being used to save, such as excel.exe or winword.exe, as our vfsmfd.sys filter driver is hooked into them and making the copies.

 

Microsoft's save to temp and rename design is described in the following Microsoft KB - under the heading "Saved Files (Same Directory as the Saved File)". It's this design that causes us to create 2 SNP files on Save instead of 1:

https://support.microsoft.com/en-gb/help/211632/description-of-how-word-creates-temporary-files

 

The following PowerShell can be used to get the average latency to the file server host being written to (wait for the command prompt to return a value, it takes several seconds):

(test-connection <FileServerHostName> -count 10 | measure-object -property responsetime -average).average

 

Take this average response time and use it in the formula below to find the expected additional latency for copying a file down from the network that goes through our kernel filter driver:

 

(Average latency in seconds) * (number of 32KB file chunks) = seconds to copy file.

 

Our vfsmfd.sys driver requests files in 32KB chunks.

• As an example with a 100MB file, you will see 3,125 chunks of 32KB in size requested if you trace the copy with a low altitude Procmon.
  • With an average latency of 20ms, the arithmetic comes out to be: 3,125 * .02 for a total of 62.5 seconds overhead in cumulative network latency.
  • This per chunk latency is inescapable due to the design of our kernel mini filter driver (vfsmfd.sys), and this is why any appropriate tuning/disabling of VEP File Elimination and/or File Restoration can speed up the Open and Save actions, by lowering the total number of file copies made from the network.

Resolution

For the “Copy to Share” channel, VEP File Elimination does not apply to saving files to network shares using Word and Excel, because of the order of operations.

Upon Save, Word/Excel write their working data to a tmp file on the network share, and we have to copy that data back in order to have a local copy of the new data that can be extracted and scanned on the endpoint.

 

Although turning File Restoration off (FileSystem.ENABLE_FILE_RESTORATION.int=0) can provide a speedup (since File Restoration makes the most copies from the network share), there are additional factors to consider if you have any endpoint policies that include response rules with an action set to "Prevent: Block" or "Prevent: User Cancel". Be aware that when a user saves a file that violates a policy that has either of these actions, the destination file will be deleted after the save.

• This is what prevents sensitive data exfiltration on the channel as you would expect, but without an SNP file to restore from the end result is the destination file is now missing on the share.
• Specifically with "Prevent: User Cancel", this will happen when the end-user either cancels the copy via the prompt, or does not respond to the prompt within the timeout ( Advanced Agent Setting of PostProcessor.OTHER_USER_RESPONSE_TIMEOUT.int ) and the setting PostProcessor.NOTIFY_WITH_CANCEL_DEFAULT_ACTION.int is set to the default of 1.

 

In either case though, the document with the user's changes is quarantined on the endpoint under "%userprofile%\My Recovered Files\FileName <%DateTimeStamp%>\, so the data can be recovered, and the response rule prompt alerts them with the path to the quarantine file.

 

The File Restoration and VEP File Elimination defaults setting are the most conservative to help ensure both the ability for our service to inspect the data contention-free and the convenience of automatic restoration of the version of the file that was opened, with the tradeoff of additional network latency. On networks with good latency in the 1 to 2 millisecond range, the cumulative latency that comes as a result of our driver requesting the files from the network in 32KB chunks is negligible. Simply put, the cumulative latency overhead increases as network latency increases.

 

​Another approach to limit impact is to pick an upper limit for Office file types for the "Copy to Share" channel using a Monitoring Filter as shown below:

 

Files that match the File Filter criteria you set in the filter above will not have a File Restoration snapshot (SNP) created at Open, nor a VEP file created at Save, as the agent does not scan the file. A single SNP file still gets created during the Save from the existing document on the share, after the tmp files have been written, and before the interim tmp file gets renamed by the editing application to replace the contents of the document that is being edited.

In one case, the issue only occurred when opening PowerPoint files (PPTX) from Network Shares when connected to VPN. Disabling Application File Access within Global Application Monitoring, just for the affected application, PowerPoint.exe in this case, resolved the issue.