Enable FTP File Mover to transfer all file types in Security Analytics

book

Article ID: 169233

calendar_today

Updated On:

Products

Security Analytics

Issue/Introduction

Like many integration providers, the FTP mover operates on a defined set of file types. The most common of those file types are controlled by checkboxes in the FTP Mover settings (see "Data Enrichment File Types" in Data Enrichment->FTP Mover->Edit). Selecting all of the available checkboxes will cause ftp mover to operate on the most common file types. However, there are some file types which are not controlled by those checkboxes in the UI. (For example, text files and pcap files.)

Resolution

To enable the FTP mover to operate on _all_available file types, one can manually edit the /etc/solera/extractor/extractord.conf file. Within that file, the file types on which integration providers operate are defined by lines of these forms: 
‚Äč
tonic_filter_$UUID=filetype1:filetype2:filetype3:... 
or 
tonic_filter_$UUID=* 

[The wildcard form means "all file types".] 

The $UUID value comes from the uuid field in the integration_providers table in postgres. Here's a psql command to look up the uuid value for the FTP Mover provider: 

[[email protected] ~]# echo "select uuid from integration_providers where name like 'FTP File%'; " | su - postgres -c 'psql dsweb'
Last login: Thu Oct 13 10:51:57 MYT 2016
stdin: is not a tty
stdin: is not a tty
                 uuid
--------------------------------------
 556d6e10-1044-4b58-ae10-5cb81e1d10a3
(1 row)


OR

If you don't want to use the database query, there is an scm script that will print the UUIDs. 

[[email protected] ~]#  scm integration_providers summary |grep 'FTP Mover'
FTP File Mover      556d6e10-1044-4b58-ae10-5cb81e1d10a3   False    FTP Mover

To force the FTP mover to operate on all file types, replace that line with a wildcard using the UUID value for FTP Mover provider as below: 

tonic_filter_ 556d6e10-1044-4b58-ae10-5cb81e1d10a3=* 

At a minimum, solera-extractord and solera-ruleEngine will need to be restarted following any such changes. A reboot may be advisable. Furthermore, manually adding a wildcard to a tonic_filter_$UUID line in the extractord.conf file is likely to make the UI "Data Enrichment File Types" checkboxes all appear to be blank. Therefore, you can manage tonic_filters with the UI, or by manually editing the extractord.conf file, but not both (this is on a per-provider basis).