Task Watcher is launched while file is being copied or written on Windows

Products

CA Automic Dollar Universe

Issue/Introduction

The File Watcher Task launches before the file has completed being copied/created, same as a Uproc with a File Resource condition with Size unchanged 000,00,01
Example of the resource log with the size unchanged condition:

2022-12-21 18:00:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 1 \| Checking resource AFR_BIG_TEMP_AFR...
2022-12-21 18:00:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 1 \| No file matching filter \[C:\temp\afr\],\[*\]
2022-12-21 18:00:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 1 \| Check KO: resource missing
2022-12-21 18:00:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 2 \| Checking resource AFR_BIG_TEMP_AFR...
2022-12-21 18:00:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 2 \| Found 1 file matching filter \[C:\temp\afr\],\[*\]
2022-12-21 18:00:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 2 \| Check condition KO for file Dollar.Universe_Application.Server_linux.64_update_6_10_102+build.0003.zip
2022-12-21 18:00:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 2 \| Check KO: resource pending
2022-12-21 18:01:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 3 \| Checking resource AFR_BIG_TEMP_AFR...
2022-12-21 18:01:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 3 \| Found 1 file matching filter \[C:\temp\afr\],\[*\]
2022-12-21 18:01:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 3 \| File Dollar.Universe_Application.Server_linux.64_update_6_10_102+build.0003.zip size has changed less than 000d 00h 01m ago
2022-12-21 18:01:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 3 \| Check condition KO for file Dollar.Universe_Application.Server_linux.64_update_6_10_102+build.0003.zip
2022-12-21 18:01:10 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 3 \| Check KO: resource pending
2022-12-21 18:01:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 4 \| Checking resource AFR_BIG_TEMP_AFR...
2022-12-21 18:01:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 4 \| Found 1 file matching filter \[C:\temp\afr\],\[*\]
2022-12-21 18:01:40 \| AFR_BIG_TEMP_AFR.00000000 \| 2924049 \| Scan Number 4 \| Check OK

Please note that this is wrong the file was still being transferred and the final file would not be present until 18:07
It seems the File Watcher / Resource Launch Formula use a standard function like DIR, to check if the file is present, but this command is not reliable when the file is being created like when doing a copy of a big file on Windows from a folder to another as the Windows Explorer already shows the final target file size and does not modify its Created or Modified Date.

Environment

Release : 6.x and 7.x

Component: Dollar Universe Application Server (Node)

OS: Windows only

Cause

This is independent to Dollar Universe, but related to how Windows handles files while being copied/transferred, the target file is seen with its final size whereas the transfer has not completed yet.

Resolution

Extract from the documentation about the Watcher Tasks:

-----

A watcher task is a particular scheduled task (with rules, dates...) that is waiting for its resource to trigger a launch.

The resource is necessarily file type but can be generic if we want to monitor the arrival of files in a directory. The behavior is the same as for a condition on a file resource with EXIST and SIZEUNCHANGED attributes.

At the beginning of its launch window, the task waits for its resource (with status Watching) and remains waiting for files with this status until the end of its launch window.

When a file arrives and its size is stabilized, the task submits the session or the Uproc that constitutes it. The launch carries the identifier of the task but has the status corresponding to the progress of its launch or execution.

------

Since Windows handles the copy of files differently than Unix/Linux, this SIZEUNCHANGED condition is fulfilled at the beginning of the transfer and not when the file has finally been completed.

To workaround this limitation on Windows, we suggest you to use a tool that verifies if the file is readable/exploitable before triggering the next uproc of the session in the Watcher Task or use a flag file for Watcher tasks instead of big files for Windows.