Posts

Patrick McHardy presents his work on a modification of netlink and nfnetlink_queue which is using memory map. One of the problem of netlink is that netlink uses regular socket I/O and data need to be copied to the socket buffer data areas before being send. This is a problem for performance. The basic concept or memory mapped netlink is to used a shared memory area which can be used by kernel and userspace. A ring buffer is set and instead of copying the data, we just move a pointer to the correct memory area and the userspace reads It is necessary to synchronize kernel and user spaces to avoid a read on a non significative area. This is done by using a area ownership. ...

Jesper presents its IP TV analyser know called IPTV-analyser. He starts the project when encountering problem in the IP TV system in the company he works for. Proprietary analyser exists but they are expensive and the tested equipment were not able to show the burstiness directly. To fix this, he started using wireshark and add it a burstiness detector. It was not enough because pcap was not scaling enough and they decide to build their own probe. One of the decisive point was the 192000€ necessary to buy the necessary probes. ...

Holger want to describe its experience when switching from monocore system to mutiticore system at Astaro Sophos. They used: RPS: Receive packet steering RFS:Receive flow steering XPS: Transmit flow steering They are using a 2.6.32 kernel and they had to backport the code but this was quite easy because the code is self-contained. irqbalance is not RPS and XPS aware and it is know to degrade performance. Holger decide then to start a new project. ...

When doing matching on iptables, the sequential test of the rules is costly. By using ipset this is possible to limit the number of matches by using the sets. For their use, they decide to use the connection mark to determine the fate of the packet. It is used to jumb on the correct chain. This logic, combined with a connectionmark set they have developed this lead to a filtering system with a really limited number of rules. In fact, this was switching from something like 10000 rules to one single rule. Ipset is doing all the classification work. The performance increase is huge as on the test system, it goes from a bandwith of 256Mb with iptables to a bandwith 1.8Gb with their system.

Ipset is now included in the kernel and that’s the main event of ipset in the previous year. József recommands to use the 6.8 version which is included in kernel 3.1. If your kernel is older, using a separately compiler ipset is recommanded. If we omit the bugfixes, a lot of of new features have been introduced sinced version 6.0. It is possible to list the sets defined on a system without getting everything which is useful when big set have been defined. ...

I gave a small presentation about a study I’ve made on connection tracking helpers. The slides are here: nfws_helper_freedom Discussion following the speech was interesting. The main subject was automatic testing of the connection tracking helpers (as well as testing the other components). Pablo Neira Ayuso came with the idea of injecting the packet inside the kernel via a mechanism similar to NFQUEUE. This would then be easy to replay traffic. An extended discussion about the subject should take place during the week.

Snet is an LSM module which treat network access. It is composed of a kernel part, a library and a tool. In the kernel, event are generated for protocol and syscall, for example tcp and listen. It is then possible through a ticket system to decide if a process has the right to the event. For example, you can tell firefox can open connections to outside. A netlink protocol is used to communicate with userspace. Thus this is possible in userspace to take the decision by issuing ticket and sending it to kernel. ...

The 8th Netfilter Workshop has been opened by Patrick McHardy in Freiburg. It is hosted by the Freiburg University. The schedule is available on the workshop wiki.

Introduction When diving in some code with a relative important size, I’ve often ask myself: where is this attribute used for this structure ? Where it is set ? Using grep is not a good answer to theses questions: you can’t guess the name of the variable of a given type and even an attribute name can be shared between multiple structures. I was in need of a semantic grep! ...

I gave a talk about Suricata entitled Suricata, rethinking IDS/IPS at Libre Software Meeting (RMLL in french). The slides can be downloaded from the RMLL website. Thanks a lot to Christophe Brocas and Mathieu Blanc for the organisation of the security track of LSM.

During one of my recent coding, I had to modify a signature file for suricata. The file was looking like this: alert pkthdr any any -> any any (msg:"SURICATA ICMPv4 unknown code"; decode-event:icmpv4.unknown_code; sid:2200024; rev:1;) alert pkthdr any any -> any any (msg:"SURICATA ICMPv4 truncated packet"; decode-event:icmpv4.ipv4_trunc_pkt; sid:2200025; rev:1;) alert pkthdr any any -> any any (msg:"SURICATA ICMPv4 unknown version"; decode-event:icmpv4.ipv4_unknown_ver; sid:2200026; rev:1;) alert pkthdr any any -> any any (msg:"SURICATA ICMPv6 packet too small"; decode-event:icmpv6.pkt_too_small; sid:2200027; rev:1;) alert pkthdr any any -> any any (msg:"SURICATA ICMPv6 unknown type"; decode-event:icmpv6.unknown_type; sid:2200028; rev:1;) alert pkthdr any any -> any any (msg:"SURICATA ICMPv6 unknown code"; decode-event:icmpv6.unknown_code; sid:2200029; rev:1;) The modification was to decrease the number behind <em<sid by 24 for each signatures. ...

Discovering the performance boost When doing some coding on both 1.0 and 1.1 branch of suricata, I’ve remarked that there was a huge performance improvement of the 1.1 branch over the 1.0 branch. The parsing of a given real-life pcap file was taking 200 seconds with 1.0 but only 30 seconds with 1.1. This performance boost was huge and I decide to double check and to study how such a performance boost was possible and how it was obtained: ...

After some time lost by trying in vain to have Kies (of Death) from Samsung oder Odin working under Virtualbox, I’ve found about the exitence of Heimdall. This software has been developped to flash firmware onto Samsung Galaxy S devices. It did work quiet easily. Upgrade procedure only requires some files download and in my case some usage of the tar command. The command line was long but simple: [bash]heimdall flash -pit s1_odin_20100512.pit –factoryfs factoryfs.rfs \ –cache cache.rfs –dbdata dbdata.rfs –param param.lfs \ –kernel zImage –modem modem.bin \ –primary-boot boot.bin –secondary-boot Sbl.bin \ –verbose[/bash] ...

I’ve gived today a presentation about Suricata at the Solutions Linux event. It was part of the security track presided by Herve Schauer. The slides are in french and are available here: 2011_sollinux_suricata

IPv6 global address The global address is used in IPv6 to communicate with the outside world. This is thus the one that is used as source for any communication and thus in a way identify you on Internet. Below is a dump of an interface configuration: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:22:15:64:42:bd brd ff:ff:ff:ff:ff:ff inet6 2a01:f123:1234:5bd0:222:15ff:fe64:42bd/64 scope global dynamic valid_lft 86314sec preferred_lft 86314sec inet6 fe80::222:15ff:fe64:42bd/64 scope link valid_lft forever preferred_lft forever</pre> The global address is here 2a01:f123:1234:5bd0:222:15ff:fe64:42bd/64. It is build by using the prefix and adding an identifier build with the hardware address. For example, here the hardware address is 00:22:15:64:42:bd and the global IPv6 address is ending with 22:15_ff:fe_64:42bd. ...

My collaboration with OISF has been announced today. This is an honor for me to join this excellent team on this wonderful project. I’ve taken a lot of pleasure in the past months contributing to the project and I’m sure the start of an official collaboration will lead to good things. The challenge is high and I will do my best to merit the trust. A big thanks to all people who congrat me for this nomination.

Suricata 1.1beta2 has brought OpenBSD to the list of supported operating system. I’m a total newbie to OpenBSD so excuse me for the lack of respect of OpenBSD standards and usages in this documentation. Here’s the different step, I’ve used to finalize the port starting from a fresh install of OpenBSD. If you want to use source taken from git, you will need to install building tools: pkg_add git libtool automake and autoconf need to be installed to. For a OpenBSD 4.8, one can run: ...

The IDS/IPS suricata has a native support for Netfilter queue. This brings IPS functionnalities to users running Suricata on Linux. Suricata 1.1beta2 introduces a lot of new features related to the NFQ mode. New stream inline mode One of the main improvement of Suricata IPS mode is related with the new stream engine dedicated to inline. Victor Julien has a great blog post about it. Multiqueue support Suricata can now be started on multiple queue by using a comma separated list of queue identifier on the command line. The following syntax: ...

Following my preceding post on suricata multithread performance I’ve decided to continue to work on the subject. By using perf-tool, I found out that when the number of detect threads was increasing, more and more time was used in a spin lock. One of the possible explanation is that the default running mode for pcap file (RunModeFilePcapAuto) is not optimal. The only decode thread take some time to treat the packets and he is not fast enough to send data to the multiple detect threads. This is triggering a lot of wait and a CPU usage increase. Following a discussion with Victor Julien, I decide to give a try to an alternate run mode for working on pcap file, RunModeFilePcapAutoFp. ...

Suricata IDS/IPS architecture is heavily using multithreading. On almost every runmode (PCAP, PCAP file, NFQ, …) it is possible to setup the number of thread that are used for detection. This is the most CPU intensive task as it does the detection of alert by checking the packet on the signatures. The configuration of the number of threads is done by setting a ratio which decide of the number of threads to be run by available CPUs (detect_thread_ratio variable). ...