Troubleshooting flex config datasource

Products

• CASB Audit
• CASB Advanced Threat Protection
• CASB Security Advanced
• CASB Gateway Advanced
• CASB Security Premium
• CASB Security Standard

Issue/Introduction

Logs are failing to process when using the following flex config.

{"datetime_format":"MMM dd yyyy HH:mm:ss z","dst_rex":"dst=(.?)[ $]","start_at_column":"50","comments_startwith":"#","bytes_rex":"bytes=(.?)[ $]","logformat":"rex","src_rex":"src=(.*?)[ $]","datetime_rex":"(?:|datetime=)(.:.?)[ $]"}

Cause

Incorrect flex config

Resolution

There are a few problems with the flex config being used.

1) CASB process "datetime_rex" first, then apply the formats defined on "datetime_format" directive, in the above example, the regular expression used, does not match the timezone section, and it truncates it. The "datetime_rex" needs to be modified accordingly (it could be something like : "datetime_rex":"(?:|datetime=)(.:.?)\s\w[ $]")

2) The log format chosen here is "REX", with REX format, there is a directive specifically for this use case which is "start_at_regex", in this example the use of "start_at_column" directive is incorrect, it is mainly designed for the other formats like syslog, .cef, .csv ..etc

3) sent and rcvd are required fields that are missing. datetime and bytes are also required fields.  

 

 

Additional Information

 

Here are a few references:

1- Flex configuration directives

2- REX Example

3- Test the format using the "Test Log Sample"