NSX Lastline components can be enhanced with customer provided intelligence that will be applied in addition to the NSX Lastline provided detection. This is known as custom intelligence, and is described in the NSX Lastline documentation here (you will need to provide your account credentials to access this page).
https://user.lastline.com/papi-doc/api/html/intel/overview.html
Using custom intelligence, customers can enhance the detection behavior by adding their own network or YARA signatures, and custom IP and Domain information.
The NSX Lastline custom intelligence API supports the following 6 types of custom intelligence:
The NSX Lastline Custom Intelligence API allows a user with multiple appliances to specify different intelligence for each appliance.
When adding intelligence entries through the API, users must select one of the following scopes, from the most general to the most specific:
Example 1:
The custom IP address denylist deployed to a Sensor is the union of all addresses in the customer’s Global IP Denylist, those in the relevant license’s IP Denylist, and those in the specific sensor’s IP Denylist.
If the same intelligence entry (e.g. a denylist IP) is present at multiple scopes that apply to the sensor, the most specific entry is selected.
Example 2:
If the IP 1.1.1.1 is in the denylist at the global scope with impact 50, and at sensor Sensor123‘s scope with impact 90, the denylist for Sensor123 will include IP 1.1.1.1 with impact 90.
Not that a single request to the add, get, del and list methods documented below always operates on a single scope.
Example 3:
If domain www.example.com is added to the Domain Denylist at the global scope, attempting to delete the domain at the license or sensor scope will have no effect: the www.example.com Domain will still be part of the Domain Denylist for all sensors, until it is removed at the global scope.
NOTE: Yara rules are applied by all Engine appliances and are thus always added on the global scope.
Please note the difference in granularity between Yara rules and the other types of custom intelligence; Yara rules apply to all the engines within a given customer license, and only apply to on-premise installations. Network signature, IP and domain intelligence can be applied globally, to all sensors in a given license, or to one or more specific sensors.
NSX Lastline uses standard RESTful APIs, so they can be used from multiple programming languages or even directly from browser plug-ins that help formulating the GET and POST requests.
To make things easier, NSX Lastline provides sample source code in the Python language, with example programs and an interactive shell that makes exploring the APIs easier.
Python is installed by default in most Linux distributions. If it is not, then please install python (version 2.7, not Python 3.x) with a command such as:
apt install python python is already the newest version. |
If installing python code directly on the NSX Lastline sensor, you will need command line access. You can either do this from the console of the sensor, or via a remote terminal session using the secure shell (ssh).
root@paris-sensor1:~# lastline_setup |
You can now login interactively with the login monitoring and the ssh password you defined above.
Assuming you have console or SSH access already, the only additional step is to make sure the Python path is defined (it’s already done for NSX Lastline appliances). This should be added to your .profile to persist over login sessions.
lastline@uclient:~/papi_client$ export PYTHONPATH=$PYTHONPATH:~lastline/papi_client |
If you ever see the error below, it’s almost certainly because $PYTHONPATH is not defined.
lastline@uclient:~/papi_client$ python scripts/papi_shell.py monitoring@paris-sensor1:~$ wget https://user.lastline.com/papi-doc/api/client/papi_client.tar.gz |
We’ve downloaded the code - now unpack the compressed archive:
monitoring@paris-sensor1:~$ tar xvfz papi_client.tar.gz |
We now have the NSX Lastline Python modules and sample source code installed and ready to use in the directory papi_client.
cd papi_client/ |
To use the custom intelligence API, users must authenticate with Account-based authentication.
monitoring@paris-sensor1:~/papi_client$ cat papi_client.ini.template |
Copy the template file to papi_client.ini and edit the username and password and url fields to match your portal credentials and manager if using NSX Lastline on-premise (don’t put any quotes around the username or password fields). If working with an on-premise manager, use verify_ssl = False.
monitoring@paris-sensor1:~/papi_client$ cp papi_client.ini.template papi_client.ini |
You have now created an authentication file for the API calls in papi_client.ini. Modifying the custom intelligence for a NSX Lastline installation (add and delete methods for each intelligence type), listing and getting feeds (list and feed methods for each intelligence type) require a user account with administrator privileges.
Viewing the configured custom intelligence (get method for each intelligence type) can be done with any user account that has access to a specified sensor or license. This permission is called “Access alerts” in the NSX Lastline Portal. When working with multiple licenses and sensors (e.g. an MSSP or complex network) you can specify which license and sensor the intelligence applies to with this parameter to the API call : 'key' : "Gxxxxxxxxxxxxxx:sensor01"
To use an API function that is querying or listing (get or list) a value or values, the HTTP method is GET, when making a change (add or delete), the HTTP method is POST. The API documentation says which method should be used for each API function.
NSX Lastline APIs require JSON input, and will generate either JSON (default) or XML as output.
We will make a web request before defining a custom signature and note there is no event generated.
We will then define a custom signature, check it is present, then generate the same request and confirm that the event appears in the NSX Lastline portal.
We’ll add a custom rule that detects HTTP requests with the exotic User-Agent: header ThunderChicken.
To try and reach our exotic avian, we’ll craft a request from a machine (note its source IP address) with curl to a web site used for NSX Lastline testing where the web request will pass by one of the sensors:
curl -A "ThunderChicken" -O http://lastlinedemo.com/demo/testfiles |
Checking network events in the NSX Lastline portal shows no event from this machine, or at least nothing that matches our chicken.
To add a custom suricata rule, we’ll need to authenticate (1), then add the signature (2). After checking the return status, we’ll list the signatures (IDS rules) (3) to make sure it’s there.
The signature we’ll add is this:
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;) |
and we’ll add two other bits of information to our rule - source : custom-ids , impact : 70
1 - Authentication
You must have installed the NSX Lastline python modules and created the papi_client.ini file in the current directory with the correct credentials for this to work. Don’t forget to define the environment variable PYTHONPATH if using a non-Lastline machine (see above).
monitoring@paris-sensor1:~/papi_client$ python scripts/papi_shell.py |
OK – so we’re connected, and there are no custom rules yet. Let’s go find some chickens!
2 - Add the signature
In [2]: client.intel.add_ids_rule([{'rule' : 'alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;)' , 'key' : "Gxxxxxxxxxxxxxx:sensor01" , 'source' : 'custom-ids' , 'impact' : 70 }]) |
We’ve added our signature with no errors. Last step is to check it’s there:
3- List signatures
In [3]: client.intel.list_ids_rule() |
OK – looks good! But does it work? Let’s find out.
Now we’re equipped with our chicken detector, let’s note our source and destination IP address and generate that request again:
monitoring@paris-sensor1:~$ hostname |
Within a couple of minutes, we should see a new event in the Events section of the portal:
Yes! We’ve seen a ThunderChicken! Let’s check the details of this scary avian (Indicators of Chicken, IoC):
We can see the message, classification, source and impact match what we put in our rule.
Looking at the captured traffic, we can see our User-Agent matches our suspicious bird:
What if we want to add multiple rules in one go? We could iterate through this process, adding one rule at a time, or if the rules share common source and impact values, we can read multiple rules from a file, like this – with a new rule, detecting deranged hounds:
monitoring@paris-sensor1:~/papi_client$ cat beagle.sig |
Custom IP Denylist:
NSX Lastline provides an example Python script that allows you to add lists of IP and Domain intelligence as comma separated files. The source code is provided in the folder examples.
If you create a sample text file in CSV format with 2 IP Addresses, a score, source and comment them, called sample_denylist.txt:
monitoring@sensor:~lastline/papi_client/examples$ cat sample_denylist.txt 5.6.7.8,30,Testsource3,comment3 |
We added one new entry, and left 2 entries unchanged.
In [6]: Result=client.intel.list_ip() |
Check the length (it should be 3) and show 1st and 2nd elements (indices 0 and 1)
In [7]: len(Result) |
Let's query the results of a specific IP address we know is there:
In [10]: client.intel.get_ip( u'3.4.5.6' ) |
If you want to delete something, you will need to specify it as a python array, in square brackets '[ element ]' rather than a single element.
In [12]: client.intel.delete_ip( [u'3.4.5.6','5.6.7.8','20.30.40.50'] )
|
Custom domain Denylist:
You can add lists of domains to the NSX Lastline domain denylist using the script described above for lists of IP Addresses, using the keyword domain rather than ip.
At the time of writing, NSX Lastline supports the Suricata 5.0.1 version which includes support for LUA scripts. NSX Lastline will continue to upgrade the version of Suricata to follow new versions, once they have proven stable. See the tutorial above for a worked example of adding Suricata rules.
Custom IDS rules:
Variables for custom IDS rules:
Reference for custom IDS rules:
Note: Currently NSX Lastline appliances utilize YARA version 3.5.0
Below are the API methods for custom YARA rule:
This is an example of how a user provided YARA rule with a priority of 100 would appear:
The NSX Lastline portal Admin tab allows Suricata IDS Rule Variables to be viewed and defined.
As noted in the section above – Scope – the level (Global to sensor specific) can be defined for each variable and value – below is a variable specific to a single sensor “France”.