Category: Suricata

Building Suricata for OpenBSD 4.9 and over

It seems OpenBSD upgrade are done to give maintenance work to the developers of third-party application. In a way, OpenBSD fight against the economic crisis: It gives jobs to developers and if you want some performance you need a powerful thus new computer.

Let’s stop bashing and be serious: Suricata was building fine on OpenBSD 4.8 but the build was failing on subsequent version. This was link with an include modification around the “socket.h” file. It is now mandatory to include “types.h” before “socket.h” to avoid compilation error. The patch 0001-Fix-OpenBSD-compilation.patch.gz fixes the build.

I’ve also updated my Building Suricata under OpenBSD post which does explain installation of dependencies which are now mandatory since Suricata 1.2.

Using AF_PACKET zero copy mode in Suricata

Victor Julien has just pushed a new feature to suricata’s git tree. It brings improvements to the AF_PACKET capture mode.

This capture mode can be used on Linux. It is the native way to capture packet. Suricata is able to use the interesting new multithreading feature provided by AF_PACKET on recent kernels: it is possible to have multiple capture threads receiving the packet of a single interface.

The commits add mmaped ring buffer support to AF_PACKET capture and also provide a zero copy mode. Mmaped ring buffer is mechanism similar to the one used by PF_RING. The kernel allocates some memory to store the packets and share this memory with the capture process. Instead of sending messages, the kernel just write to the shared memory and the process capture reads it. This is less consuming in term of CPU ressource and helps to increase the capture rate. But the main avantage of this technique is that the capture process can treat the packets without making a copy and this saves a lot of time

To activate this features, you need a Suricata compiled from latest git and you need to modify some entries in your suricata.yaml file. You have to tell suricata that you want to activate the mmap feature. For example to activate the feature on eth0, you have to add ‘use-mmap’ to your configuration:
[code]
af-packet:
– interface: eth0
use-mmap: yes
[/code]

You can then run Suricata with the command:

suricata -c suricata.yaml --af-packet=eth0

This setup will not activate the zero copy feature which is currently dependant of the running mode. You will need to activate the worker mode to enable zero copy. To do so, run Suricata with a command similar to this one:

suricata -c suricata.yaml --af-packet=eth0 --runmode=workers

This code should provide an interesting performance boost to the AF_PACKET capture system. I’ve no number to provide now but I will be happy to hear some if you make some tests

Ecosystem of Suricata

Suricata is an IDS/IPS engine. To build a complete solution, you will need to use other tools.

The following schema is a representation of a possible software setup in the case Suricata is used as IDS or IPS on the network. It only uses opensource components:

Suricata is used to sniff and analyse the traffic. To detect malicious traffic, it uses signatures (or rules). You can download a set of specialised rules from EmergingThreats.

To analyse the alerts generated by Suricata, you will need an interface such as, for example, snorby.

But this interface will need to have an access to the data. This is done by storing the alerts in a database. The feed of this database can be done by barnyard2 which take the files outputted by suricata in unified2 and convert and send them to a database (or to some other format).

If you want counter-measure to be taken (such as blocking for a moment attacker at the firewall level), you can use snortsam.

More complex setup can include interactions with a SIEM solution.

Acquisition systems and running modes evolution of Suricata

Some new features have recently reach Suricata’s git tree and will be available in the next development release. I’ve worked on some of them that I will describe here.

Multi interfaces support and new running modes

Configuration update

IDS live mode in suricata (pcap, pf_ring, af_packet) now supports the capture on multiple interfaces. The syntax of the YAML configuration file has evolved and it is now possible to set per-interface variables.

For example, it is possible to define pfring configuration with the following syntax:

pfring:
  - interface: eth4
    threads: 8
    cluster-id: 99
    cluster-type: cluster_flow
  - interface: eth1
    threads: 2
    cluster-id: 98
    cluster-type: cluster_round_robin

This set different parameters for the eth4 and eth2 interfaces. With that configuration, it the user launches suricata with

suricata -c suricata.yaml --pfring

it will be listening on eth4 and with 8 threads receiving packets on eth4 with a flow based load balancing and 2 threads on eth3.

If you want to run suricata on a single interface, simply do:

suricata -c suricata.yaml --pfring=eth4

This syntax can be used with the new AF_PACKET acquisition module describe below.

New running modes

The running modes have been extended by a new running mode available for pfring and af_packet which is called workers. This mode starts a configurable number of threads which are doing all the treatment from packet acquisition to logging.

List of running modes

Here is the list of current running modes:

  • auto: Multi threaded mode (available for all packet acquisition modules)
  • single: Single threaded mode (available in pcap, pcap file, pfring, af_packet)
  • workers: Workers mode (available in AF_PACKET and pfring)
  • autofp: Multi threaded mode. Packets from each flow are assigned to a single detect thread.

af_packet support

Suricata now supports acquisition via AF_PACKET. This linux packet acquisition socket has recently evolved and it supports now load balancing of the capture of an interface between userspace sockets. This module can be configured like show at the start of this post. It will run on almost any Linux but you will need a 3.0 kernel to be able to use the load balancing features.

suricata -c suricata.yaml --af-packet=eth4

OISF brainstorming: planning phase 3 (take 3)

GEO IP

Idea is to add a keyword that would be used to interact with GEOIP database (free at least) and be able to use it to detect things like control canal. For example, an IRC server in an non common country is certainly a control canal.

Live ruleset swap

A must have! This is vital for critical environnement. This is very costly in memory and this should be an option to avoid exploding low memory boxes.

Qosmos integration / API for data exchange

Bringing protocol analysis is an interesting point as it will help to increase performance and accuracy of the engine. Knowing the protocol permit to only run protocol related rules to flow of that protocol. And this avoid to have false detection by running the rules on bad protocol. OpenDPI technology and Qosmos technology integration is discussed. A common API is needed to be able to use both systems.

Global shared flowvars

Global flow var will permit to change the way we build rules. Not being constrained anymore to stream variable will increase the power of rules.

Host/app/OS table import

Idea is to load host type from file to be able to tune the host setting precisely.

IPFIX support

IPFIX support as entry or output could bring some advantages.

Conclusion

Matt Jonkman and Victor Julien will now summarize the input and publish on OISF website the planned features for phase 3 based on discussion about priority of the tasks that have been held.

OISF brainstorming: planning phase 3 (take 2)

DNS fast flux/anomaly detection

The idea is to detect malware and other things by collecting the DNS request and their answer and detecting anomaly. For example, if an host is making a lot of request to a domain.

First part of the job on Suricata is to log all requests and their answer. Then analysis can occurs in the database.

File extraction

This is a work under progress linked with a third party contract. It permit to store exchanged files on disk for some application level protocol. It is possible to say: “store the file, if the content type is different from the extension”. File extraction works currently on HTTP. It focus on POST request to detect uploaded file.

Time-based counters

This aims at detection of regular behaviours which are often linked with command control connection. For example, triggering an alert if a specific DNS request is done every five minutes.

HTTP keyword improvement

HTTP keywords improvement are discussed, some specific keywords could be added to avoid the cost of pcre. A two parameters header match is suggested to support all possible keyword.

Output

Discussion is about logging the normalised application level content. Currently, only the packet triggering the lart is loggued and thus the information about why suricata has logged is lost. This is thus interesting to log the reconstructed application level message to permit the analyst to analyse the reason of the alert.

Regarding the output module, it could be interesting to adds support for CEE logging which would be able to support the resulting composite alert. For performance reason, a barnyard output of this composite alert is interesting. It may be needed to suggest some change to support this application level alert.

Oisf brainstorming: planning phase 3 (take 1)

Performance improvement

As shown by Victor’s latest work on performance counters, there is a lot of work that can be done to improve performance. They are currently good but there is place for improvement. Proposal to provide off-loading or clustering is done. This is heavily discussed but as pointed out by Victor, it will be more interesting to do this in the next phase. Phase 3 should focus in improvement of current code. This will permit to use the upcoming Suricata killing features like global flow variable.

SSL preprocessor

Following the recent certicate authority attacks, a SSL preprocessor which is able to detect blacklist certificate and other things will be really interesting. It could also detect certificate property change whan connection to host or similar things.

Decryption is not seen by the participants of the webex has important. Decryption would be performance killer without accelerator. Thus a limitation to certificate analysis is at first an interesting target

IP and DNS reputation

The idea is too exchange IP reputation between sensors to protect themself against offensive hosts as soon as it has been identified on one sensor. This requires exchange between nodes and this is already done by other projects. Doing it alone in Suricata would be to big and thinking about adding a way to interact easily with this project could be a first step.

Matt Jonkman: development avancement

Phase 2 development is almost over now. Among the completed major features:

  • Multithread
  • protocol discovery
  • smb logging
  • HTTP logging
  • flowvars

One of the advantage of Suricata over Snort is protocol discovery combined to HTTP parsing by libhtp. It provides a huge improvement over Snort as a lot of bad flow are using HTTP on non standard ports.

Victor Julien: Development status

Work has started in september 2007. The work depends on some externel library like multithread of input handling library. The main external depedency is libhtp which is initally developped by Ivan Ristic.

The development is managed in a single git repository. Victor is the only one with commit right. The review are done by Victor and cross review are made by developpers.

Work unit for developers are tasks which are written by Victor and describe a specific task to do. This task are mainly done by OISF funded developers. Some simpler task are let to the comunity and everyone can help with this.

To offload Victor’s load, subsystem mainteners are nominated:

  • Eric Leblond: packet acquisition
  • Anoop Saldanha: detection part

They will have freedom on the way to improve the subsystem they are in charge.

The development is currently done with two branches (1.0 which is bug fix only and master). Victor is currently not happy with this and would like to switch to time-based release. This is discussed as this can be difficult for company to deal with frequent updates. A funding of maintenance by companies could help to keep the current working system.

Peter Manev is in charge of the QA. There is a lot of work to do in this area. Unit test is currently good but there is a lot of work to do to improve detection of regression.

Performance has been improved in 1.1 with a focus on efficient algorithm.

CUDA support needs help. The performance is still lower with than without and thus this really need developer power!

Performance profiling have been added recently by Victor and it shows clearly that work is needed on this.

To sum up, the two main areas where help will be most than welcome are QA and performance profiling.

Matt Jonkman: introduction speech

Matt presents the goal of the OISF brainstorming session:

  • Make a status of the foundation
  • Grabbing new ideas

The session will be interactive and anybody is invited to participate through physical intendance or webex.

The foundation is non-profitable and aim at building a powerful engine for us all. OISF is member og the HOST program and happily supported by some industrials.

Foundation business

Matt fills he can not give enough times to the foundation due to his work at EmergingThreat and propose to hire a General Manager that would take care of finding the funding and administrative part.

The funding renewal was smallest than planned and the work on getting funding must be increased. Situation is not bad but improvement could be interesting.

Potential major projects

Universal Rules Language

This is a highly difficult problem. It has been wanted by major actors for years but no successfull and clean approach has been found.

Commercial support

A lot of industrials want to have access to commercial support to integrate Suricata in their product. Developers can not do the job but they can form some support people to have them become experts.

OpenDPI/Qosmos integration

The idea is to add support for multiple protocols to be able to provide high level rules for a lot more of protocols.

Scada

Digital bound give a copyright usage to the foundation to integrade a preprocessor into Suricata.

Snortsam integration

Snortsam is now a plugin to barnyard and this made it more easy to use.