Plaisirs et désillusions du monde moderne
My collaboration with OISF has been announced today. This is an honor for me to join this excellent team on this wonderful project. I’ve taken a lot of pleasure in the past months contributing to the project and I’m sure the start of an official collaboration will lead to good things. The challenge is high and I will do my best to merit the trust.
A big thanks to all people who congrat me for this nomination.
Suricata 1.1beta2 has brought OpenBSD to the list of supported operating system. I’m a total newbie to OpenBSD so excuse me for the lack of respect of OpenBSD standards and usages in this documentation.
Here’s the different step, I’ve used to finalize the port starting from a fresh install of OpenBSD.
If you want to use source taken from git, you will need to install building tools:
pkg_add git libtool
automake and autoconf need to be installed to. For a OpenBSD 4.8, one can run:
pkg_add autoconf-2.61p3 automake-1.10.3
For a OpenBSD 5.[01], one can run:
pkg_add autoconf-2.61p3 automake-1.10.3p3
For OpenBDS 5.2:
pkg_add autoconf-2.61p3 automake-1.10.3p6
Autoconf 2.61 is know to work, some other versions triggers a compilation failure.
Then you can simply clone the repository and run autogen:
git clone git://phalanx.openinfosecfoundation.org/oisf.git cd oisf export AUTOCONF_VERSION=2.61 export AUTOMAKE_VERSION=1.10 ./autogen.sh
Before running configure, you need to add the dependencies:
pkg_add gcc pcre libyaml libmagic libnet-1.1.2.1p0
Now, we’re almost done and we can run configure:
CPPFLAGS="-I/usr/local/include" CFLAGS="-L/usr/local/lib" ./configure --prefix=/opt/suricata
You can now run make and make install to build and install suricata.
The IDS/IPS suricata has a native support for Netfilter queue. This brings IPS functionnalities to users running Suricata on Linux.
Suricata 1.1beta2 introduces a lot of new features related to the NFQ mode.
One of the main improvement of Suricata IPS mode is related with the new stream engine dedicated to inline. Victor Julien has a great blog post about it.
Suricata can now be started on multiple queue by using a comma separated list of queue identifier on the command line. The following syntax:
suricata -q 0 -q 1 -c /etc/suricata.yaml
will start a suricata listening to Netfilter queue 0 and 1.
The option has been added to improve performance of Suricata in NFQ mode. One of the observed limitation is the number of packets per second that can be sent to a single queue. By being able to specify multiple queues, this is possible to increase performances. The best impact is on multicore systems where it adds scalability.
This feature can be used with the queueâ€balance option the NFQUEUE target which was added in Linux 2.6.31. When using −−queue−balance x:x+n instead of −−queue−num, packets are then balanced across the given queues and packets belonging to the same connection are put into the same nfqueue.
One of the difficulty of IPS usage is to build an adapted firewall ruleset. Following my blog post on this issue, I’ve decided to implement most of the existing mode in Suricata.
When running in NFQ inline mode, it is possible now to use a simulated non-terminal NFQUEUE verdict by using the ‘repeat’ mode.
This permit to send all needed packet to suricata via this a rule:
iptables -I FORWARD -m mark ! --mark $MARK/$MASK -j NFQUEUE
And below, you can have your standard filtering ruleset.
To activate this mode, you need to set mode to ‘repeat’ in the nfq section.
nfq: mode: repeat repeat_mark: 1 repeat_mask: 1
This option uses the NF_REPEAT verdict instead of using a standard NF_ACCEPT verdict. The effect is that the packet is sent back at the start of the table where the queuing decision has been taken. As Suricata has put on the packet the mark $repeat_mark with respect to the mask $repeat_mask, when the packet reaches again the iptables NFQUEUE rules it does not match anymore because of the mark and the ruleset floowing this rule is used.
The ‘mode’ option in nfq section can have two other values:
The idea behind the route option is that a program using NFQ can issues a verdict that has for effect to send the packet to an another queue. This is thus possible to chain softwares using NFQ.
To activate this option, you have to use the following syntax:
nfq: mode: route route_queue: 2
There is not much usage of this option. One mad mind could think at chaining suricata and snort in IPS mode 😉
A new keyword nfq_set_mark has been added to the rules option.
This is an enhancement of the NFQ mode. If a packet matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask specified in the option during the verdict.
The usage is really simple. In any rules, you can add the nfq_set_mark keyword to specify the mark to put on a packet and what is the the
pass tcp any any -> any 80 (msg:"TCP port 80"; sid:91000004; nfq_set_mark:0x80/0x80; rev:1;)
If you are not familiar with the advanced network capabilities, you may wondering how it can be used. Nefilter mark can be used to modify the handling of a packet by the network stack, ie routing, quality of service, Netfilter. Thus, the concept is the following, you can use Suricata to detect a suspect packet and decide to change the way it is handled by the network stack.
Thanks to the CONNMARK target, you can modify the way all packets of the connection are handled.
Among the possibilities:
Following my preceding post on suricata multithread performance I’ve decided to continue to work on the subject.
By using perf-tool, I found out that when the number of detect threads was increasing, more and more time was used in a spin lock. One of the possible explanation is that the default running mode for pcap file (RunModeFilePcapAuto) is not optimal. The only decode thread take some time to treat the packets and he is not fast enough to send data to the multiple detect threads. This is triggering a lot of wait and a CPU usage increase. Following a discussion with Victor Julien, I decide to give a try to an alternate run mode for working on pcap file, RunModeFilePcapAutoFp.
The architecture of this run mode is different. A thread is in charge of the reading of the file and the treatment of packets is done in a pool of threads (from decode to output). The augmentation of the power of decoding and the limitation of the ratio decode/detect would possibly bring some scalability.
The following graph is a comparison of the Auto mode and the FP mode on the test system described in the previous post (A 24 threads/core server parses a 6.1Go pcap file). It displays the number of packets per second in function of the number of threads:
The performance difference is really interesting. The FP mode shows a increase of the performance with the number of threads. This is far better than the Auto run mode where performance decrease with the number of threads.
As pointed out in a discussion on the OISF-users mailing list, multithread tuning has a real impact on performance. The result of the tests I’ve done are significant but they only apply to the parsing of a big pcap file. You will have to tune Suricata to see how to take the best of it on your system.
Suricata IDS/IPS architecture is heavily using multithreading. On almost every runmode (PCAP, PCAP file, NFQ, …) it is possible to setup the number of thread that are used for detection. This is the most CPU intensive task as it does the detection of alert by checking the packet on the signatures. The configuration of the number of threads is done by setting
a ratio which decide of the number of threads to be run by available CPUs (detect_thread_ratio variable).
A discussion with Florian Westphal at NFWS 2010 convince me that it was necessary for performance to tune with more granularity the thread and CPU link. I thus decide to improve this part of the code in Suricata and I’ve recently submitted a serie of patches that enable a fine setting.
I’ve been able to run some tests on a decent server:
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
This is a dual 6-core CPUs with hyperthreading activated and something like 12 Go of memory. From a Linux point of view, it has 24 identical processors.
I was not able to produce a decent testing environnement (read able to inject traffic) and all tests have been made on the parsing of a 6.1Go pcap file captured on a real network.
I’ve first tested my modification and soon arrive to the conclusion that limiting the number of threads was a good idea. Following Victor Julien suggestion, I’ve played with the detect_thread_ratio variable to see its influence on the performance. Here’s the graph of performance relatively to the ratio for the given server:
It seems that the 0.125 value (which correspond to 3 threads) is on this server the best value.
If we launch the test with ratio 0.125 more than one time, we can see that the performance vary between 64s and 50s with a mean of 59s. This means the variation is about 30% and the running can not be easily predicted.
It is now time to look at the results we can obtain by tuning the affinity. I’ve setup the following affinity configuration which run 3 threads on selected CPU (for our reader 3 = 0.125 * 24):
cpu_affinity:
- management_cpu_set:
cpu: [ 0 ] # include only these cpus in affinity settings
- receive_cpu_set:
cpu: [ 1 ] # include only these cpus in affinity settings
- decode_cpu_set:
cpu: [ "2" ]
mode: "balanced"
- stream_cpu_set:
cpu: [ "0-4" ]
- detect_cpu_set:
#cpu: [ 6, 7, 8 ]
cpu: [ 6, 8, 10 ]
#cpu: [ 6, 12, 18 ]
mode: "exclusive" # run detect threads in these cpus
threads: 3
prio:
low: [ "0-4" ]
medium: [ "5-23" ]
default: "medium"
- verdict_cpu_set:
cpu: [ 0 ]
prio:
default: "high"
- reject_cpu_set:
cpu: [ 0 ]
prio:
default: "low"
- output_cpu_set:
cpu: [ "0" ]
prio:
default: "medium"
and I’ve played with detect_cpu_set setting on the cpu variable. By using CPU set coherent with the hardware architecture, we manage to have result with a very small variation between run:
Thus we stabilize the best performance by remaining on the same hardware CPU and avoiding hyperthread CPUs. This also explain the difference between the run of the first tests. The only setting was the number of threads and we can encounter any of the setup (same or different hardware CPU, running two threads on a core with hyperthreading) or even flip during one test. That’s why performance vary a lot between tests run.
Next step for me will be to run perf tool on suricata to estimate where the bottleneck is. More infos to come !
During Nefilter Workshop 2008, we had an interesting discussion about the fact that NFQUEUE is a terminal decision. This has some strong implication and in particular when working with an IPS like suricata (or snort-inline at the time of the discussion): the IPS must received all packets routed by the gateway and can only issue a terminal DROP or ACCEPT verdict. It thus take precedence over all subsequent rules in the ruleset: any ACCEPT rules before the IPS rules will remove packets from IPS analysis and in the other way, any decision after the IPS rules will be ignored.
First question is where to put the IPS rules. From a Netfilter point of view, the NFQUEUE rule for suricata should be put as first rule of the FORWARD filter chain but this is not possible as NFQUEUE is a terminal rule. A classic trick is to use PREROUTING mangle to put the rule in but this is not a good choice as destination NAT has not yet been done: The IPS will not be able to see the real target of packet and will then not be able to use things like OS or server type declaration. Thus, the current best-bet decision seems to use FORWARD mangle.
This solution is not solving the main issue, ACCEPT can be used in the mangle table and thus having a NFQUEUE rules in the last place will not work. Here, we’ve got two possibilities:
The NFWS 2008 was covering the second case and Patrick McHardy has proposed an interesting solution. It is not very well known but a NFQUEUE verdict can take three values:
Patrick proposes to use the NF_REPEAT feature: the IPS rule is put in first place and the IPS issues a NF_REPEAT verdict instead of NF_ACCEPT for accepted packets. As NF_REPEAT will trigger a infinite loop, we need a way to distinguish the packet that have already been treated by the IPS. This can easily be done by putting a mark on the packet during the verdict. This feature is supported by NFQUEUE since its origin and the IPS could do it easily
(the only condition for this solution is to be able to dedicate one bit of the mark to this system).
With this system, the necessary rule to have suricata intercept packet is the following:
iptables -A FORWARD ! -i lo -m mark ! --mark 0x1/0x1 -j NFQUEUE
Here, we’ve got mark and mask set to 1. By adding this simple rule at the top of the FORWARD filter chain, we obtain a ruleset which combine easily the inspection of all packets in the IPS with the traditional filtering ruleset.
I’ve sent a patch to modify suricata in this way. In this new nfq.repeat_mode, suricata issues a NF_REPEAT verdict instead of a NF_ACCEPT verdict and put a mark ($MARK) with respect to a mask ($MASK) on the handled packet.
Let suppose now we can modify the rule generation system. We thus have all flexibitlity needed to buid a custom ruleset which combine the filtering and the IPS task.
Let’s recall our main target: they want the IPS to analyse all packets going through the gateway. By analysing this in the Netfilter scope, we could formulate this in this way: We want to send all packets which are accepted in the FORWARD to the IPS. This is equivalent to replace all the ACCEPT verdict by the action of sending the packet to the IPS. To do this we can simply use a custom user chain:
iptables -N IPS iptables -A IPS -j NFQUEUE --queue-balance 0:1
Then, we replace all ACCEPT rules by a target sending the packet to the IPS chain (use -j IPS instead of -j ACCEPT).
Note: Some reader will have seen that I’m using in the NFQUEUE rules the queue-balance option which specifies a range of queue to use. Please note that packets from the same connections are sent to the same queue and that, at the time fo the writing, Suricata has patches which are currently under review and wich add multiqueue support.
The interest of using a custom chain is that we can defined things like exception or special treatment in the chain. For example, to ignore a specific computer (1.2.3.4 in the example), we can do:
iptables -I IPS -d 1.2.3.4 -j ACCEPT
Some may object that we don’t get every packets because we send only accepted packets to the IPS. My answer is that this not the IPS role to treat this ones. This is the role of the firewall to send alert on blocked packet. And this is the role of the SIEM to combine firewall logs with IPS information. If you really want all packet to be send to the IPS, then repeat mode is your friend.
One real issue with the setup here described is the handling of multiple programs using the NFQUEUE. The previous method can only applied to one program, as sending to NFQUEUE will be terminal.
Here, we have two solutions. First one is to used the NFQ_REPEAT method on program different than the IPS and using NFQUEUE. The packet will reach after some iteration a -j IPS rule and we will have the wanted result.
An other method is to use the queue routing capabilities of the NFQUEUE. The verdict is a 32 bit integer but only the first 16 bit are used by the verdict. The other 16 bit are used if not null to indicate on which queue the packet have to sent after the verdict by the current program. This method is elegant but requires a support of the feature by the involved programs.
Suricata is a next generation IDS/IPS engine developed by the Open Information Security Foundation.
This article describes the installation, setup and usage of Suricata with CUDA support on a Ubuntu 10.04 64bit. For 32 bit users, simply remove 64 occurances where you find them.
You need to download both Developper driver and Cuda driver from nvidia website. I really mean both because Ubuntu nvidia drivers are not working with CUDA.
I’ve first downloaded and installed CUDA toolkit for Ubuntu 9.04. It was straightforward:
sudo sh cudatoolkit_3.0_linux_64_ubuntu9.04.run
To install the nvidia drivers, you need to disconnect from graphical session and close gdm. Thus I’ve done a CTRL+Alt+F1 and I’ve logged in as normal user. Then I’ve simply run the install script:
sudo stop gdm
sudo sh devdriver_3.0_linux_64_195.36.15.run
sudo modprobe nvidia
sudo start gdm
After a normal graphical login, I was able to start working on suricata build
I describe here compilation of 0.9.0 source. To do so, you can get latest release from OISF download page and extract it to your preferred directory:
wget http://openinfosecfoundation.org/download/suricata-0.9.0.tar.gz
tar xf suricata-0.9.0.tar.gz
cd suricata-0.9.0
Compilation from git should be straight forward (if CUDA support is not broken) by doing:
git clone git://phalanx.openinfosecfoundation.org/oisf.git
cd oisf
./autogen.sh
Configure command has to be passed options to enable CUDA:
./configure –enable-debug –enable-cuda –with-cuda-includes=/usr/local/cuda/include/ –with-cuda-libraries=/usr/local/cuda/lib64/ –enable-nfqueue –prefix=/opt/suricata/ –enable-unittests
After that you can simply use
make
sudo make install
Now you’re ready to run.
Let’s first check, if previous step were correct by running unittests:
sudo /opt/suricata/bin/suricata -uUCuda
It should display a bunch of message and finish with a summary:
==== TEST RESULTS ====PASSED: 43FAILED: 0======================
Now, it is time to configure Suricata. To do so we will first install configuration file in a standard location:
sudo mkdir /opt/suricata/etc
sudo cp suricata.yaml classification.config /opt/suricata/etc/
sudo mkdir /var/log/suricata
wget http://www.emergingthreats.net/rules/emerging.rules.tar.gzcd /opt/suricata/etc/sudo tar xf /home/eric/src/suricata-0.9.0/emerging.rules.tar.gz
default-rule-path: /etc/suricata/rules/
default-rule-path: /opt/suricata/etc/rules/
classification-file: /opt/suricata/etc/classification.config
sudo tcpdump -s0 -i br0 -w Desktop/br0.pcap
sudo /opt/suricata/bin/suricata -c /opt/suricata/etc/suricata.yaml -r /home/eric/Desktop/br0.pcap
#mpm-algo: b2gmpm-algo: b2g_cuda
time sudo /opt/suricata/bin/suricata -c /opt/suricata/etc/suricata.yaml -r /home/eric/Desktop/br0.pcap 2>/tmp/out.log