Day: December 5, 2012

About Suricata and a kernel oops in AF_PACKET

Introduction

Kernel oops have been reported by some users running Suricata with AF_PACKET multiple thread capture activated. This is due to a bug I’ve introduced in AF_PACKET when fixing an other bug.

Which kernel not to use with Suricata in AF_PACKET mode

The following kernel version will surely crash if Suricata or any other program is used with AF_PACKET capture with multiple capture threads:

  • Linux 3.2.30 to 3.2.33
  • Linux 3.4.12 to 3.4.18
  • Linux 3.5.5 to 3.5.7
  • Linux 3.6.0 to 3.6.6

If only one capture thread is used there is no risk of crash. If you are running a vulnerable kernel, your configuration should looks like:

af-packet:
  - interface: eth0
    # Number of receive threads (>1 will crash with bad kernel)
    threads: 1

Some explanations

AF_PACKET capture is one of my favorite capture method on Linux with Suricata. It is mainline and it offers really good performance with latest kernel. For example, this is deployed on one of our box and run at 10Gbps speed on non-expensive hardware.
This speed is achieved by using load-balanced capture. This feature allows to have multiple thread/process listening to the same interface. The load-balancing is made by the kernel.
This feature has been developed by David Miller and is available since Linux 3.1.

In summer 2012, I’ve worked on adding AF_PACKET IPS mode and I’ve discovered a kernel bug which was causing a packet sending loop in the IPS code. I’ve proposed a fix af_packet: don’t emit packet on orig fanout group. The patch has reached mainline with Linux 3.6. As it was fixing a real problem it was propagated to most Linux stable branches. Some distributions, like Ubuntu, have also backported the patch to their official kernel.

But the patch was buggy and some Suricata users have reported kernel oops. I’ve fixed the bug af-packet: fix oops when socket is not present and the patch will be available in Linux 3.7.
The kernel stable team has incorporated this patch in their subsequent releases so most stable branches but 3.5 don’t suffer anymore of this problem.

Note

Ubuntu Quantal has a patched kernel since at least 3.5.0-25.