Suricata

I’ve been invited by SaLUG to Lecce to give some talks during their Geek Evening. I’ve done a talk on nftables and one of suricata. Lecce by night The nftables talk was about the motivation behind the change from iptables. Here are the slides: Nftables The talk on Suricata was explaining the different feature of Suricata and was showing how I’ve used it to make a study of SSH bruteforce. ...

Introduction I’ve been playing lately on analysis SSH bruteforce caracterization. I was a bit frustrated of just getting partial information: ulogd can give information about scanner settings suricata can give me information about software version sshd server logs shows username But having username without having the password is really frustrating. So I decided to try to get them. Looking for a SSH server honeypot, I did find kippo but it was going too far for me by providing a fake shell access. So I’ve decided to build my own based on paramiko. ...

Some progress on the JSON side Suricata 2.0-rc2 is out and it brings some progress on the JSON side. The logging of SSH protocol has been added: and the format of timestamp has been updated to be ISO 8601 compliant and it is now named timestamp instead of time. ...

Iptables and suricata as IPS Building a Suricata ruleset with iptables has always been a complicated task when trying to combined the rules that are necessary for the IPS with the firewall rules. Suricata has always used Netfilter advanced features allowing some more or less tricky methods to be used. For the one not familiar with IPS using Netfilter, here’s a few starting points: IPS receives the packet coming from kernel via rules using the NFQUEUE target The IPS must received all packets of a given flow to be able to handle detection cleanly The NFQUEUE target is a terminal target: when the IPS verdicts a packet, it is or accepted (and leave current chain) ...

Log analysis experiment I’ve been playing lately with logstash using data from the ulogd JSON output plugin and the Suricata full JSON output as well as standard system logs. Ulogd is getting Netfilter firewall logs from Linux kernel and is writing them in JSON format. Suricata is doing the same with alert and other traces. Logstash is getting both log as well as sytem log. This allows to create some dashboard with information coming from multiple sources. If you want to know how to configure ulogd for JSON output check this post. For suricata, you can have a look at this one. ...

Introduction Perf is a great tool to analyse performances on Linux boxes. For example, perf top will give you this type of output on a box running Suricata on a high speed network: Events: 32K cycles 28.41% suricata [.] SCACSearch 19.86% libc-2.15.so [.] tolower 17.83% suricata [.] SigMatchSignaturesBuildMatchArray 6.11% suricata [.] SigMatchSignaturesBuildMatchArrayAddSignature 2.06% suricata [.] tolower@plt 1.70% libpthread-2.15.so [.] pthread_mutex_trylock 1.17% suricata [.] StreamTcpGetFlowState 1.10% libc-2.15.so [.] __memcpy_ssse3_back 0.90% libpthread-2.15.so [.] pthread_mutex_lock The functions are sorted by CPU consumption. Using arrow key it is possible to jump into the annotated code to see where most CPU cycles are used. ...

Introduction logstash an opensource tool for managing events and logs. It is using elasticsearch for the storage and has a really nice interface named Kibana. One of the easiest to use entry format is JSON. Suricata is an IDS/IPS which has some interesting logging features. Version 2.0 will feature a JSON export for all logging subsystem. It will then be possible to output in JSON format: HTTP log DNS log TLS log File log IDS Alerts For now, only File log is available in JSON format. This extract meta data from files transferred over HTTP. ...

Not usual setup can lead to surprise The 5th of December 2012, I’ve setup suricata in AF_PACKET IPS mode between a WiFi interface and an Ethernet interface. The result was surprising as it was leading to a crash after some time: The issue was linked with the defrag option of AF_PACKEt fanout. I’ve proposed a patch the 7th Dec 2012 and after a discussion with David Miller and Johannes Berg, Johannes has proposed a better patch which was included in official tree. So the problem is fixed for kernel superior or equal to 3.7. ...

Suricata and Netfilter can be better friend as they are doing some common work like decoding packet and maintaining flow table. In IPS mode, Suricata is receiving raw packet from libnetfilter_queue. It has to made the parsing of this packet but this kind of thing has also been done by kernel. So it should be possible to avoid to duplicate the work. ...

A huge work Suricata 1.4 has been released December 13th 2012 and it has been a huge work. The number of modifications is just impressing: 390 files changed, 25299 insertions(+), 11982 deletions(-) The following video is using gource to display the evolution of Suricata IDS/IPS source code between version 1.3 and version 1.4. It only displays the modified files and do not show the files existing at start. A collaborative work A total of 11 different authors have participated to this release. The following graph generated by gitstats shows the number of lines of code by author: ...

Introduction Kernel oops have been reported by some users running Suricata with AF_PACKET multiple thread capture activated. This is due to a bug I’ve introduced in AF_PACKET when fixing an other bug. Which kernel not to use with Suricata in AF_PACKET mode The following kernel version will surely crash if Suricata or any other program is used with AF_PACKET capture with multiple capture threads: Linux 3.2.30 to 3.2.33 Linux 3.4.12 to 3.4.18 Linux 3.5.5 to 3.5.7 Linux 3.6.0 to 3.6.6 If only one capture thread is used there is no risk of crash. If you are running a vulnerable kernel, your configuration should looks like: ...

The naive approach would consider that an IDS is just taking packet and doing a lot of matching on it. In fact, this is not at all what is happening. An IDS/IPS like Suricata is in fact rebuilding the data stream and in case of known protocols it is even normalizing the data stream and providing keyword which can be used to match on specific field of a protocol. Let’s say, we a rule to match on a HTTP request where method is GET and the URL is “/download.php”. ...

lstlisting is a convenient way to display code when using latex. It has no definition for suricata rules language and I’ve cooked one: \lstdefinelanguage{suricata} {morekeywords= {alert, tcp, http, tls, ip, ipv4, ipv4, drop, pass, sid, priority, rev, classtype, threshold, metadata, reference, tag, msg, content, uricontent, pcre, ack, seq, depth, distance, within, offset, replace, nocase, fast\_pattern, rawbytes, byte\_test, byte\_jump, sameip, ip\_proto, flow, window, ftpbounce, isdataat, id, rpc, dsize, flowvar, flowint, pktvar, noalert, flowbits, stream\_size, ttl, itype, icode, tos, icmp\_id, icmp\_seq, detection\_filter, ipopts, flags, fragbits, fragoffset, gid, nfq\_set\_mark, tls.version, tls.subject, tls.issuerdn, tls.fingerprint, tls.store, http\_cookie, http\_method, urilen, http\_client\_body, http\_server\_body, http\_header, http\_raw\_header, http\_uri, http\_raw\_uri, http\_stat\_msg, http\_stat\_code, http\_user\_agent, ssh.protoversion, ssh.softwareversion, ssl\_version, ssl\_state, byte\_extract, file\_data, dce\_iface, dce\_opnum, dce\_stub\_data, asn1, filename, fileext, filestore, filemagic, filemd5, filesize, l3\_proto, luajit}, otherkeywords={ipv4-csum, tcpv4-csum, tcpv6-csum, udpv4-csum, udpv6-csum, icmpv4-csum, icmpv6-csum, decode-event, app-layer-event, engine-event, stream-event}, sensitive=true, morecomment=[l]{//}, morecomment=[s]{/*}{*/}, morestring=[b]", } To use it, you can simply add this code at start of your tex file and you can then use it: ...

Introduction Some times ago, I’ve blogged about new IPS features in Suricata 1.1 and did not find at the time any killer application of the nfq_set_mark keyword. When using Suricata in Netfilter IPS mode, this keyword allows you to set the Netfilter mark on the packet when a rule match. This mark can be used by Netfilter or by other network subsystem to differentiate the treatment to apply to the packet. ...

Introduction I’ve been working for the past few days on a new Suricata feature. It is available in Suricata 1.4rc1. Suricata can now listen to a unix socket and accept commands from the user. The exchange protocol is JSON-based and the format of the message has been done to be generic and it is described in this commit message. An example script called suricatasc is provided in the source and installed automatically when updating Suricata. ...

A new Suricata IPS mode Suricata IPS capabilities are not new. It is possible to use Suricata with Netfilter or ipfw to build a state-of-the-art IPS. On Linux, this system has not the best throughput performance. Patrick McHardy’s work on netlink: memory mapped I/O should bring some real improvement but this is not yet available. I’ve thus decided to do an implementation of IPS based on AF_PACKET (read raw socket). The idea is based on one of the snort’s running mode. It peers two network interfaces and all packets received from one interface are sent to the other interface (if a signature with drop keyword does not fired on the packet). This requires to dedicate two network interfaces for Suricata but this provide a simple bridge system. As suricata is using latest AF_PACKET features (read load balancing), it was possible to build something really promising. ...

Suricata TLS support Victor Julien has just merged to main tree a branch containing some interesting new TLS related features. They have been contributed by me and Jean-Paul Roliers. This patchset introduces TLS logging and brings some new keywords to Suricata engine. Here’s the list of all TLS related keywords that are available in latest Suricata git: tls.version: match on version of protocol tls.subject: match on subject of certificate tls.issuerdn: match on issuer DN of certificate tls.fingerprint: match on SHA1 fingerprint of certificate tls.store: store the certificate on disk You will find detailed explanation below. ...

Introduction Since the beginning of July 2012, OISF team is able to access to a server where one interface is receiving some mirrored real European traffic. When reading “some”, think between 5Gbps and 9.5Gbps constant traffic. With that traffic, this is around 1Mpps to 1.5M packet per seconds we have to study. The box itself is a standard server with the following characteristics: CPU: One Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (16 cores counting Hyperthreading) Memory: 32Go capture NIC: Intel 82599EB 10-Gigabit SFI/SFP+ The objective is simple: be able to run Suricata on this box and treat the whole traffic with a decent number of rules. With the constraint not to use any non official system code (plain system and kernel if we omit a driver). ...

It seems OpenBSD upgrade are done to give maintenance work to the developers of third-party application. In a way, OpenBSD fight against the economic crisis: It gives jobs to developers and if you want some performance you need a powerful thus new computer. Let’s stop bashing and be serious: Suricata was building fine on OpenBSD 4.8 but the build was failing on subsequent version. This was link with an include modification around the “socket.h” file. It is now mandatory to include “types.h” before “socket.h” to avoid compilation error. The patch 0001-Fix-OpenBSD-compilation.patch.gz fixes the build. ...

Victor Julien has just pushed a new feature to suricata’s git tree. It brings improvements to the AF_PACKET capture mode. This capture mode can be used on Linux. It is the native way to capture packet. Suricata is able to use the interesting new multithreading feature provided by AF_PACKET on recent kernels: it is possible to have multiple capture threads receiving the packet of a single interface. The commits add mmaped ring buffer support to AF_PACKET capture and also provide a zero copy mode. Mmaped ring buffer is mechanism similar to the one used by PF_RING. The kernel allocates some memory to store the packets and share this memory with the capture process. Instead of sending messages, the kernel just write to the shared memory and the process capture reads it. This is less consuming in term of CPU ressource and helps to increase the capture rate. But the main avantage of this technique is that the capture process can treat the packets without making a copy and this saves a lot of time ...