NSX Network Detection and Response - Denylist or enhance detection using custom signature intelligence (Yara and Suricata rules)
Article ID: 323965
Updated On:
Products
Issue/Introduction
Introduction
NSX Lastline components can be enhanced with customer provided intelligence that will be applied in addition to the NSX Lastline provided detection. This is known as custom intelligence, and is described in the NSX Lastline documentation here (you will need to provide your account credentials to access this page).
https://user.lastline.com/papi-doc/api/html/intel/overview.html
Using custom intelligence, customers can enhance the detection behavior by adding their own network or YARA signatures, and custom IP and Domain information.
Types of custom intelligence
The NSX Lastline custom intelligence API supports the following 6 types of custom intelligence:
- Denylist IPs: (sensors): An NSX Lastline Sensor will generate events when it detects traffic to these IPv4 addresses.
- Denylist Domains: (sensors): An NSX Lastline Sensor will generate events when it detects DNS requests for these domains or traffic to IPs resolved from these domains.
- IDS Rules: (sensors): An NSX Lastline Sensor will generate events when these IDS rules match against the monitored network traffic. In addition to the rules themselves, the custom intelligence API supports variables for the custom IDS rules as well as references for documenting the rules.
- IDS Rule variables: (sensors): The custom IDS rules provided by a customer may include variables. The values for these variables are then provided as a separate feed. This allows a customer to, for example, define a rule at the global scope and specialize it on each sensor with different variable values defined at the sensor scope. These can also be set using the NSX Lastline portal user interface.
- IDS Rule references: (sensors): The custom IDS rules provided by a customer may include shorthand references to resources such as the CVE archive. The custom intelligence API allows users to define the mapping between reference keywords (such as “cve”) and the URL prefix used to resolve the reference URL (such as “https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-”). These mappings are treated just like another entity type, although they can only be defined at the global scope.
- Yara Rules: (engines): The NSX Lastline Engine will generate alerts when an analyzed objects matches one of the given rules. You can only add custom YARA rules to on-premise installations of NSX Lastline, not to hosted instances.
Scope of custom intelligence
The NSX Lastline Custom Intelligence API allows a user with multiple appliances to specify different intelligence for each appliance.
When adding intelligence entries through the API, users must select one of the following scopes, from the most general to the most specific:
- Global Scope: The intelligence entries in the global scope apply to all of the user’s appliances (all appliances covered by all licenses)
- License Scope: The intelligence entries in the license scope apply to all a user’s appliances configured with a specified NSX Lastline license (all appliances for a single license)
- Sensor Scope: The intelligence entries in the sensor scope apply only to a specific sensor (single sensor)
The intelligence feed used by a NSX Lastline Sensor combines intelligence entries in the relevant global, license and sensor scopes.
Example 1:
The custom IP address denylist deployed to a Sensor is the union of all addresses in the customer’s Global IP Denylist, those in the relevant license’s IP Denylist, and those in the specific sensor’s IP Denylist.
If the same intelligence entry (e.g. a denylist IP) is present at multiple scopes that apply to the sensor, the most specific entry is selected.
Example 2:
If the IP 1.1.1.1 is in the denylist at the global scope with impact 50, and at sensor Sensor123‘s scope with impact 90, the denylist for Sensor123 will include IP 1.1.1.1 with impact 90.
Not that a single request to the add, get, del and list methods documented below always operates on a single scope.
Example 3:
If domain www.example.com is added to the Domain Denylist at the global scope, attempting to delete the domain at the license or sensor scope will have no effect: the www.example.com Domain will still be part of the Domain Denylist for all sensors, until it is removed at the global scope.
NOTE: Yara rules are applied by all Engine appliances and are thus always added on the global scope.
Please note the difference in granularity between Yara rules and the other types of custom intelligence; Yara rules apply to all the engines within a given customer license, and only apply to on-premise installations. Network signature, IP and domain intelligence can be applied globally, to all sensors in a given license, or to one or more specific sensors.
Installation
NSX Lastline uses standard RESTful APIs, so they can be used from multiple programming languages or even directly from browser plug-ins that help formulating the GET and POST requests.
To make things easier, NSX Lastline provides sample source code in the Python language, with example programs and an interactive shell that makes exploring the APIs easier.
Install python
Python is installed by default in most Linux distributions. If it is not, then please install python (version 2.7, not Python 3.x) with a command such as:
|
apt install python python is already the newest version. |
Install NSX Lastline Python API shell and examples
Installing on NSX Lastline sensor
If installing python code directly on the NSX Lastline sensor, you will need command line access. You can either do this from the console of the sensor, or via a remote terminal session using the secure shell (ssh).
|
root@paris-sensor1:~# lastline_setup |
You can now login interactively with the login monitoring and the ssh password you defined above.
Installing on a different machine
Assuming you have console or SSH access already, the only additional step is to make sure the Python path is defined (it’s already done for NSX Lastline appliances). This should be added to your .profile to persist over login sessions.
|
lastline@uclient:~/papi_client$ export PYTHONPATH=$PYTHONPATH:~lastline/papi_client |
If you ever see the error below, it’s almost certainly because $PYTHONPATH is not defined.
|
lastline@uclient:~/papi_client$ python scripts/papi_shell.py monitoring@paris-sensor1:~$ wget https://user.lastline.com/papi-doc/api/client/papi_client.tar.gz |
We’ve downloaded the code - now unpack the compressed archive:
|
monitoring@paris-sensor1:~$ tar xvfz papi_client.tar.gz |
We now have the NSX Lastline Python modules and sample source code installed and ready to use in the directory papi_client.
|
cd papi_client/ |
Authentication
To use the custom intelligence API, users must authenticate with Account-based authentication.
|
monitoring@paris-sensor1:~/papi_client$ cat papi_client.ini.template |
Copy the template file to papi_client.ini and edit the username and password and url fields to match your portal credentials and manager if using NSX Lastline on-premise (don’t put any quotes around the username or password fields). If working with an on-premise manager, use verify_ssl = False.
|
monitoring@paris-sensor1:~/papi_client$ cp papi_client.ini.template papi_client.ini |
You have now created an authentication file for the API calls in papi_client.ini. Modifying the custom intelligence for a NSX Lastline installation (add and delete methods for each intelligence type), listing and getting feeds (list and feed methods for each intelligence type) require a user account with administrator privileges.
Viewing the configured custom intelligence (get method for each intelligence type) can be done with any user account that has access to a specified sensor or license. This permission is called “Access alerts” in the NSX Lastline Portal. When working with multiple licenses and sensors (e.g. an MSSP or complex network) you can specify which license and sensor the intelligence applies to with this parameter to the API call : 'key' : "Gxxxxxxxxxxxxxx:sensor01"
API input and output
To use an API function that is querying or listing (get or list) a value or values, the HTTP method is GET, when making a change (add or delete), the HTTP method is POST. The API documentation says which method should be used for each API function.
NSX Lastline APIs require JSON input, and will generate either JSON (default) or XML as output.
Tutorial – adding and testing a custom Suricata signature
Objective
We will make a web request before defining a custom signature and note there is no event generated.
We will then define a custom signature, check it is present, then generate the same request and confirm that the event appears in the NSX Lastline portal.
We’ll add a custom rule that detects HTTP requests with the exotic User-Agent: header ThunderChicken.
Generate web request before adding signature
To try and reach our exotic avian, we’ll craft a request from a machine (note its source IP address) with curl to a web site used for NSX Lastline testing where the web request will pass by one of the sensors:
|
curl -A "ThunderChicken" -O http://lastlinedemo.com/demo/testfiles |
Checking network events in the NSX Lastline portal shows no event from this machine, or at least nothing that matches our chicken.
Adding a custom signature
To add a custom suricata rule, we’ll need to authenticate (1), then add the signature (2). After checking the return status, we’ll list the signatures (IDS rules) (3) to make sure it’s there.
The signature we’ll add is this:
|
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;) |
and we’ll add two other bits of information to our rule - source : custom-ids , impact : 70
1 - Authentication
You must have installed the NSX Lastline python modules and created the papi_client.ini file in the current directory with the correct credentials for this to work. Don’t forget to define the environment variable PYTHONPATH if using a non-Lastline machine (see above).
|
monitoring@paris-sensor1:~/papi_client$ python scripts/papi_shell.py |
OK – so we’re connected, and there are no custom rules yet. Let’s go find some chickens!
2 - Add the signature
|
In [2]: client.intel.add_ids_rule([{'rule' : 'alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;)' , 'key' : "Gxxxxxxxxxxxxxx:sensor01" , 'source' : 'custom-ids' , 'impact' : 70 }]) |
We’ve added our signature with no errors. Last step is to check it’s there:
3- List signatures
|
In [3]: client.intel.list_ids_rule() |
OK – looks good! But does it work? Let’s find out.
Generate web request after adding signature
Now we’re equipped with our chicken detector, let’s note our source and destination IP address and generate that request again:
|
monitoring@paris-sensor1:~$ hostname |
Within a couple of minutes, we should see a new event in the Events section of the portal:
Yes! We’ve seen a ThunderChicken! Let’s check the details of this scary avian (Indicators of Chicken, IoC):
We can see the message, classification, source and impact match what we put in our rule.
Looking at the captured traffic, we can see our User-Agent matches our suspicious bird:
Adding multiple rules
What if we want to add multiple rules in one go? We could iterate through this process, adding one rule at a time, or if the rules share common source and impact values, we can read multiple rules from a file, like this – with a new rule, detecting deranged hounds:
|
monitoring@paris-sensor1:~/papi_client$ cat beagle.sig |
Adding custom IP intelligence
Custom IP Denylist:
- add_ip(): Add or update one or more IPs in the custom IP denylist
- get_ip(): Get an IP’s entry from the custom IP denylist
- del_ip(): Remove one or more IPs from the custom IP denylist
- list_ip(): List the IPs in the custom IP denylist
- feed_ip(): Get the IP denylist feed as it would be served to a sensor
NSX Lastline provides an example Python script that allows you to add lists of IP and Domain intelligence as comma separated files. The source code is provided in the folder examples.
If you create a sample text file in CSV format with 2 IP Addresses, a score, source and comment them, called sample_denylist.txt:
|
monitoring@sensor:~lastline/papi_client/examples$ cat sample_denylist.txt 5.6.7.8,30,Testsource3,comment3 |
We added one new entry, and left 2 entries unchanged.
|
In [6]: Result=client.intel.list_ip() |
Check the length (it should be 3) and show 1st and 2nd elements (indices 0 and 1)
|
In [7]: len(Result) |
Let's query the results of a specific IP address we know is there:
|
In [10]: client.intel.get_ip( u'3.4.5.6' ) |
If you want to delete something, you will need to specify it as a python array, in square brackets '[ element ]' rather than a single element.
|
In [12]: client.intel.delete_ip( [u'3.4.5.6','5.6.7.8','20.30.40.50'] )
|
Adding custom Domain intelligence
Custom domain Denylist:
- add_domain(): Add or update one or more domains in the custom domain denylist
- get_domain(): Get a domain’s entry from the custom domain denylist
- del_domain(): Remove one or more domains from the custom domain denylist
- list_domain(): List the domains in the custom domain denylist
- feed_domain(): Get the domain denylist feed as it would be served to a sensor
You can add lists of domains to the NSX Lastline domain denylist using the script described above for lists of IP Addresses, using the keyword domain rather than ip.
Adding custom IDS (Suricata) rules
At the time of writing, NSX Lastline supports the Suricata 5.0.1 version which includes support for LUA scripts. NSX Lastline will continue to upgrade the version of Suricata to follow new versions, once they have proven stable. See the tutorial above for a worked example of adding Suricata rules.
Custom IDS rules:
- add_ids_rule(): Add or update one or more custom IDS rules
- get_ids_rule(): Get a custom IDS rule
- del_ids_rule(): Remove one or more custom IDS rules
- list_ids_rule(): List the custom IDS rules
- feed_ids_rule(): Get the feed of custom IDS rules as it would be served to a sensor
Adding custom IDS (Suricata) variables
Variables for custom IDS rules:
- add_ids_rule_variable(): Add or update one or more variables for custom IDS rules
- get_ids_rule_variable(): Get the value of a variable for custom IDS rules
- del_ids_rule_variable(): Remove one or more variables for custom IDS rules
- list_ids_rule_variable(): List the variables for custom IDS rules
- feed_ids_rule_variable(): Get the feed of variables for custom IDS rules as it would be served to a sensor
Adding custom IDS (Suricata) rule references
Reference for custom IDS rules:
- add_ids_rule_reference(): Add or update one or more references for custom IDS rules
- get_ids_rule_reference(): Get the value of a reference for custom IDS rules
- del_ids_rule_reference(): Remove one or more reference for custom IDS rules
- list_ids_rule_reference(): List the references for custom IDS rules
Adding custom YARA rules
Note: Currently NSX Lastline appliances utilize YARA version 3.5.0
Below are the API methods for custom YARA rule:
- add_yara_rule(): Add or update one or more Yara rules
- get_yara_rule(): Get a Yara rule previously added to the system
- del_yara_rule(): Remove one or more Yara rules
- list_yara_rule(): List the Yara rules previously added to the system
- feed_yara_rule(): Get the Yara rule feed as it would be served to an Engine appliance
This is an example of how a user provided YARA rule with a priority of 100 would appear:
NSX Lastline user interface for custom rule variable intelligence
The NSX Lastline portal Admin tab allows Suricata IDS Rule Variables to be viewed and defined.
As noted in the section above – Scope – the level (Global to sensor specific) can be defined for each variable and value – below is a variable specific to a single sensor “France”.
Additional Information
Feedback
