Posts

I did not write any article on coccigrep since the 1.0 release. Here is an update on what has been added to the software since that release. C++ support Coccinelle has a basic C++ support which can be activated by using the –cpp flag in coccigrep. Patches information The -L -v options on command line will display a description of the match available on the system. $ coccigrep -L -v set: Search where a given attribute of structure 'type' is set * Confidence: 80% * Author: Eric Leblond <eric@regit.org> * Arguments: type, attribute * Revision: 2 For the developer, this is obtained from structured comments put at the start of the cocci file: ...

Some new features have recently reach Suricata’s git tree and will be available in the next development release. I’ve worked on some of them that I will describe here. Multi interfaces support and new running modes Configuration update IDS live mode in suricata (pcap, pf_ring, af_packet) now supports the capture on multiple interfaces. The syntax of the YAML configuration file has evolved and it is now possible to set per-interface variables. For example, it is possible to define pfring configuration with the following syntax: ...

GEO IP Idea is to add a keyword that would be used to interact with GEOIP database (free at least) and be able to use it to detect things like control canal. For example, an IRC server in an non common country is certainly a control canal. Live ruleset swap A must have! This is vital for critical environnement. This is very costly in memory and this should be an option to avoid exploding low memory boxes. ...

DNS fast flux/anomaly detection The idea is to detect malware and other things by collecting the DNS request and their answer and detecting anomaly. For example, if an host is making a lot of request to a domain. First part of the job on Suricata is to log all requests and their answer. Then analysis can occurs in the database. File extraction This is a work under progress linked with a third party contract. It permit to store exchanged files on disk for some application level protocol. It is possible to say: “store the file, if the content type is different from the extension”. File extraction works currently on HTTP. It focus on POST request to detect uploaded file. ...

Performance improvement As shown by Victor’s latest work on performance counters, there is a lot of work that can be done to improve performance. They are currently good but there is place for improvement. Proposal to provide off-loading or clustering is done. This is heavily discussed but as pointed out by Victor, it will be more interesting to do this in the next phase. Phase 3 should focus in improvement of current code. This will permit to use the upcoming Suricata killing features like global flow variable. ...

Phase 2 development is almost over now. Among the completed major features: Multithread protocol discovery smb logging HTTP logging flowvars One of the advantage of Suricata over Snort is protocol discovery combined to HTTP parsing by libhtp. It provides a huge improvement over Snort as a lot of bad flow are using HTTP on non standard ports.

Work has started in september 2007. The work depends on some externel library like multithread of input handling library. The main external depedency is libhtp which is initally developped by Ivan Ristic. The development is managed in a single git repository. Victor is the only one with commit right. The review are done by Victor and cross review are made by developpers. Work unit for developers are tasks which are written by Victor and describe a specific task to do. This task are mainly done by OISF funded developers. Some simpler task are let to the comunity and everyone can help with this. ...

Matt presents the goal of the OISF brainstorming session: Make a status of the foundation Grabbing new ideas The session will be interactive and anybody is invited to participate through physical intendance or webex. The foundation is non-profitable and aim at building a powerful engine for us all. OISF is member og the HOST program and happily supported by some industrials. Foundation business Matt fills he can not give enough times to the foundation due to his work at EmergingThreat and propose to hire a General Manager that would take care of finding the funding and administrative part. ...

Patrick (aka kaber) has just made available his work on memory mapped netlink. Both the kernel and the libmnl part are available on git.kernel.org. You can pull kernel code other net-next tree: git pull git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nl-mmap-2.6.git Libmnl code can be fetched: git clone git://git.kernel.org/pub/scm/linux/kernel/git/kaber/libmnl-mmap.git Once done a NETLINK_MMAP kernel compilation option is then available via make config. Documentation is available in the Linux tree. It is in the file: Documentation/networking/netlink_mmap.txt

The Netfilter workshop being a developer conference, I’ve decided to presente an introduction to the coccinelle tool. Coccinelle is a program matching and transformation engine for the C language which is used in many place and among them in the Linux kernel. It is able to perform C clever modification in the code. If you ever had to modify multiple code files following an API change, I invite you to have a look at the slides or my Coccinelle for the newbie page. I’ve also presented my coccigrep tool which is a easy to use semantic grep. ...

Jesper’s IPTables::libiptc is a perl module which allow you to modify Netfilter rules from Perl. He’s the maintener and this is available on CPAN. It currently supports up-to iptables 1.4.10 (version 0.51 of IPTables::libiptc). It dynamically load xtables.so and libiptc.so to access to iptables feature. It is fast as it does not suffer of iptables limitation (which is running modification one by one). Performance are quite good: it takes only 16 sec to generate and implement a 80000 rules ruleset (which is quite good compare to the 42h hours that would be take by direct iptables calls) ...

Patrick presents one work that is aiming at getting rid of the second tuple in the connection tracking. This second tuple is only necessary when NAT is used. idea is not new but at the time the ct-extention where not available and thus it would not be possible to add it when needed. Patrick has done most of the work but there is still a missing point which is the hash function. It has to be symetrical: hash_func(src,dst) = hash_func(dst, src) and it must be very fast to avoid slowdown of the conntrack. ...

We have been ignoring the fact that NAT could have some interest in IPv6 during the latest 5 years. IPv6 will not fix everything and it may be time to reconsider NAT. There is some reasons for that: Dynamic IPv6 prefixes: some ISP decide to not give fixed address to people Server load balancing, DMZ Uplink Balancing (multi-homing): this is one of the most important reason. IPv6 client can handle multiple addresses but you may want not having your user to choose their internet output. ...

Pablo is presenting is work on protocol classification. As you may not have guess, nfgrep is not using regular expression but a descriptive language. The basic architecture is the following: developped layer-7 filter in userspace filter is passed to a tool that generates byte-code it loads the byte-code to the kernel via nfnetlink The kernel does the classification nfgrep match can then be used to select or mark the flow In userspace, nfgrep and libnfgrep can be used to interact with the system. There’s also a nfgrep-test to validate filter before sending them. ...

Cyberoam team presents their work on active active cluster. They’ve done a 2 nodes active active setup, with a primary and an auxiliary sytem. The primary take care of load balancing. The setup is using virtual MAC addresses. To avoid split-brain problem, the primary take all decisions by always treating the SYN packet. It also transfer the NAT, marks to the auxiliary thanks to a module. This is done via a module called ipt_SYNDATA. It is placed in PREROUTING ...

At times it is necessary to flush UNREPLIED connection tracking entries for connectionless protocols if there are NAT rules involved. For example this is the case when a ipsec or a ppp connection goes up. Without doing that the connection are not correctly NATed because the topology change has not been taken into account. Doing this in userspace with the conntrack-tools was taking long like minutes on some setup. They thus decide to put in kernel space and this is now only taking milliseconds instead of minutes. ...

There is a fixed number of connection tracking entries. When reaching the maximum, new connections are simply dropped. Default maximum size is ridicully too low like using 20Mbytes oon a 12GB memory computer. Kernel syslog message “nf_conntrack: table full, dropping, packet” is not correct because packet have just no state relatively to conntrack. Usually they get blocked by invalid rules but an adapted ruleset could let them go through. One other problem is that adjusting the connection tracking size does not change the hash size. This results in longer search because conntrack has often to go through a list. ...

Jan starts its presentation by talking about its Distro Availability Matrix of Netfilter tech page. It contains the software and their versions in a lot of distributions. Next subject is the discussion about maintaining translations of iptables man page. The team is international and could translate in a few language the man pages. But the question is about finding volunteers in the long term. Jan is alright with taking in charge the synchronization of translation. Any volunteers for translation is welcome. ...

Reverse Path filtering is currently only implemented in IPv4. Eric Leblond sends a patch to add support for IPv6 but it was refused by David Miller who, among other points, wanted to get rid of rp_filter and would like to see it in the Netfilter code. Reverse patch filter implementation is a single function called fib_validate_source. Looking at the problem, it seem relatively simple to implement because, it is just to reverse source and destination and then get the output interface. if it match with the incoming interface, then this is ok. ...

I just gave a presentation to explain that it is necessary to implement carefully reverse path filtering in IPv4 and IPv6. More to come later.