Implementing multiple file check logic in S3 jobs - Automic IG S3 agent

Products

• Automic Automation

Issue/Introduction

Learn how to configure the Automic IG S3 agent to monitor or process multiple files using Regular Expressions (RegEx) instead of standard wildcards. This article also addresses how to handle logic for failing jobs when multiple files match the criteria.

Environment

• Automic Automation v26
• IG S3 agent v24.5.0 and higher

Cause

Standard OS wildcard globbing (e.g., *.txt) is interpreted as a literal filename by the S3 agent and is not supported for pattern matching.

Resolution

To implement "multiple file" monitoring or processing, Regular Expressions must be enabled within the job definition.

Configuration Steps:

1. Navigate to the S3 Job definition in the Automic Web Interface (AWI).
2. Locate the job's settings form and select the Use Regex checkbox.
3. In the File Name (or Source File Path) field, enter a Java-compatible RegEx pattern.
   • Match all CSV files: .*\.csv
   • Match all text files: .*\.txt
   • Match specific prefixes: cdm.*\.csv
4. Handling Subdirectories: If files are located in a subfolder, do not include the path in the Regex field. Use the Query Param field:
   • File Name: .*\.csv
   • Query Param: prefix=folder/subfolder/

Failing Jobs on Multiple Matches: Currently, the S3 agent processes all files matching the pattern. To fail a job when more than one file is found:

• Use a post-processing script to read the report log.
• Count the number of files listed.
• Set the job to a non-zero return code if the count is greater than one.
• Note: A defect is under review to improve native handling of this logic. Subscribe to this article to be updated on the fix status (Reference: 275360).

Additional Information

• This functionality is supported for Exist, Download, Copy, Delete, and Monitor jobs.