New AF_PACKET IPS mode in Suricata

A new Suricata IPS mode

Suricata IPS capabilities are not new. It is possible to use Suricata with Netfilter or ipfw to build a state-of-the-art IPS. On Linux, this system has not the best throughput performance. Patrick McHardy’s work on netlink: memory mapped I/O should bring some real improvement but this is not yet available.

I’ve thus decided to do an implementation of IPS based on AF_PACKET (read raw socket). The idea is based on one of the snort’s running mode. It peers two network interfaces and all packets received from one interface are sent to the other interface (if a signature with drop keyword does not fired on the packet). This requires to dedicate two network interfaces for Suricata but this provide a simple bridge system. As suricata is using latest AF_PACKET features (read load balancing), it was possible to build something really promising.

Victor Julien has commited my implementation which is now available in current git tree and will be part of Suricata 1.4beta1.

Configuration

You need to dedicate two network interfaces for this mode. The configuration is made via some new configuration variable available in the description of an AF_PACKET interface.

For example, the following configuration will create a Suricata acting as IPS between interface eth0 and eth1:

af-packet:
  - interface: eth0
    threads: 1
    defrag: yes
    cluster-type: cluster_flow
    cluster-id: 98
    copy-mode: ips
    copy-iface: eth1
    buffer-size: 64535
    use-mmap: yes
  - interface: eth1
    threads: 1
    cluster-id: 97
    defrag: yes
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: eth0
    buffer-size: 64535
    use-mmap: yes

Basically, we’ve got an af-packet configuration with two interfaces. Interface eth0 will copy all received packets to eth1 because of the copy-* configuration variable:

    copy-mode: ips
    copy-iface: eth1

The configuration on eth1 is symetric:

    copy-mode: ips
    copy-iface: eth0

There is some important facts to consider when setting up this mode:

  • The implementation of this mode is dependent of the zero copy mode of AF_PACKET. Thus you need to set use-mmap to yes on both interface.
  • MTU on both interfaces have to be equal: the copy from one interface to the other is direct and packet bigger then the MTU will be dropped by kernel.
  • Set different values of cluster-id on both interfaces to avoid conflict.

The copy-mode variable can take the following values:

  • ips: the drop keyword is honored and matching packet are dropped.
  • tap: no drop occurs, Suricata act as a bridge

Please note, that mode can be different on the peered interfaces.

Current limitation

One other important point is that the threads variable must be equal on both interface. This is due to the fact the peering is done at the socket level: each packet received on a socket is sent to a peered socket listening to the peered interface.

At last but not least, you need at least Linux kernel 3.6 (or 3.4.12) to be able to set a threads value higher than 1. There is a bug in prior kernel that causes an infinite loop (packet being sent from an interface and reread, …). If you can’t wait or can’t made a full upgrade of your kernel you can use my patch which will be part of Linux 3.6 and of Linux 3.4.12.

Running it

Running Suricata in AF_PACKET IPS mode is straight forward. Once the configuration has been updated, you can simply run:

suricata -c /etc/suricata/suricata.yaml --af-packet

Please note that is possible to have normal IDS interface running simultaneously. For example, eth3 could be added to the af-packet configuration and used a regular interface.

Some implementation details

You can find some additional technical information in the commit.

As mentioned below, the peering is done at the socket level. In Suricata, each capture thread is opening a socket and is binding it to the cluster-id of the interface.
Once a thread is started, it is peered with an thread listening to the copy-iface interface which socket will be used as a send interface.
This system permit to avoid locking issue in workers running mode as only one thread will write to the peered socket at a time because we have only one packet treated by a capture thread at a time.

We need to reuse the interface for a simple reason: if a new dedicated socket was to be used, the packet sent to this socket would be received by Suricata read socket instead of being ignored. Linux kernel implement the natural feature of avoiding to send back packet to the sending socket.

This behavior was implemented in Linux kernel for regular socket but this was not the case for clustered sockets. This is why at least Linux 3.6 or my patch is needed to be able to use a value of threads bigger than one.

Conclusion

This mode is new in Suricata and is is really promising. It has suffered limited testing in high bandwidth environment. So, as usual, feedback is more than welcome. Don’t hesitate to send me a mail or to ask question on Suricata Mailing lists.

30 thoughts on “New AF_PACKET IPS mode in Suricata”

  1. Hí,

    suricatasc error:

    ~/suricata$ sudo suricatasc
    Traceback (most recent call last):
    File “/usr/local/bin/suricatasc”, line 30, in
    socket.connect(SOCKET_PATH)
    File “/usr/lib/python2.7/socket.py”, line 224, in meth
    return getattr(self._sock,name)(*args)
    socket.error: [Errno 2] No such file or directory

    Best Regards,

  2. OK, did you activate the feature ? You have to put in your suricata.yaml something like:

    unix-command:
      enabled: yes
    
  3. Hi,

    I tried it, but suricata won’ start, my config part:
    af-packet:
    – interface: eth1 # Number of receive threads (>1 will enable experimental flow pinned # runmode)
    threads: 1
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    copy-mode: ips
    copy-iface: eth2

    When I run suricata (suricata c /etc/suricata/suricata-debian.yaml –af-packet)
    I receive this error:
    29/5/2013 – 12:39:17 – – Adding interface eth1 from config file
    29/5/2013 — 12:39:17 – – Using 1 live device(s).
    29/5/2013 — 12:39:17 – – Enabling mmaped capture on iface eth1
    29/5/2013 — 12:39:17 – – AF_PACKET TAP mode activated eth1->eth2
    29/5/2013 — 12:39:17 – – Using flow cluster mode for AF_PACKET (iface eth1)
    29/5/2013 — 12:39:17 – – Using defrag kernel functionality for AF_PACKET (iface eth1)
    29/5/2013 — 12:39:17 – – Enabling zero copy mode by using data release call
    29/5/2013 — 12:39:17 – – [ERRCODE: SC_ERR_AFP_CREATE(190)] – Threads number not equals
    29/5/2013 — 12:39:17 – – [ERRCODE: SC_ERR_RUNMODE(187)] – Some IPS capture threads did not peer.

    A new entry appears in kern.log:
    Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev- instead.

    versions:
    suricata 1.4.1
    kernel: 3.8
    system: debian sid

    What could be the problem?

    Laszlo

  4. Does the AF_PACKET IPS mode in Suricata just support the bridge mode, does it ok in route mode

  5. Could you please post a valid IPtables configuration that works with this setup? while Suricata is reporting:

    11/23/2013-13:42:20.196528 [wDrop] [**] [1:1:1] facebook is blocked [**] [Classification: Potential Corpo
    rate Privacy Violation] [Priority: 1] {TCP} 192.168.X.X:XXXX -> XX.XX.XX.XX:80

    Packets are not dropped, they still pass to the internal network.

    My IPTables Config:

    Chain INPUT (policy DROP)
    target prot opt source destination
    ACCEPT all — anywhere anywhere
    ACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
    ACCEPT icmp — anywhere anywhere limit: avg 1/sec burst 1
    LOG icmp — anywhere anywhere limit: avg 1/sec burst 1 LOG level warning prefix “PING-DROP:”
    DROP icmp — anywhere anywhere
    ACCEPT tcp — anywhere anywhere multiport dports ssh
    ACCEPT tcp — anywhere anywhere tcp dpt:domain
    ACCEPT udp — anywhere anywhere udp dpt:domain
    ACCEPT tcp — anywhere anywhere tcp dpt:http
    ACCEPT udp — 192.168.1.15 anywhere multiport dports radius:tdp-suite
    ACCEPT gre — anywhere anywhere

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    Thank you!

  6. Hello Joseph. This AF_PACKET IPS mode is complete separated from firewall stack. It establishes a software bridge between two interfaces by copying packet from one interface to the other (and reverse).
    So packet are not seen by the firewall stack.

  7. Thanks, Regit, I’ll keep my iptables rules then… The problem is although I see the wDrop statements in fast.log based on website content blocking rules, I still can access the blocked content, while with NFQUEUE it worked like a charm. I’m only getting 45MBps throughtput with a dual-core intel atom 1.86GHz and 2GiB DDR3 1333Mhz in NFQUEUE mode, so I wanted to test AF_PACKET in order to improve performance.. Suggestions are welcome.

  8. Same here:

    12/11/2013-20:34:45.359743 [wDrop] [**] [1:2100498:7] GPL ATTACK_RESPONSE id check returned root [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 82.165.177.154:80 -> 192.168.1.33:56836

    traffic passes.

    Any Ideas?

  9. No I did not. Pass the dunce cap.

    Statically assigned and working now. I assumed ‘auto’ meant it was active based upon the note in yaml: “# auto will use inline mode in IPS mode”

    Was expecting Drop still…is there any reason why the fast log shows a “wDrop?”

    Thanks for your help!

  10. I also had “inline yes” streaming from the beginning and traffic still passes. “wDrop” logs show, despite not dropping.

  11. I am also having the problem “wDrop” logs showing, despite not dropping. Inline is set to “yes” in suricata.yaml. If I create a “reject” rule instead of a “drop” rule, the reject seems to prevent the packet being passed, where as “drop” still passes.

    Is this a bug?

  12. Hello Chris,

    Do you have a bridge setup ? You don’t need a bridge in AF_PACKET IPS mode. Suricata will be in charge of transferring packets from one interface to the other.

  13. Regit, thanks for the reply.

    I did not setup a bridge manually or intentionally. “brctl show” does not list any bridges.

    I have eth0 set to DHCP. I have eth1 setup to “Share to other computers”. I have a router setup to DHCP
    So it goes INTERNET > cable modem > eth0 of IPS > eth1 of IPS> linksys router > LAN. The LAN can get to INTERNET even when IPS is not running, oddly. So maybe the packets not “dropping” is related to this. But again, packets will be “rejected” properly by the IPS when I switch a “drop rule” to be a “reject rule”.

    I am thinking /etc/NetworkManager is messing things up, since I set “managed” to true. I am going to try to flush all and set the IPS route and gateways up manually, but I am not sure how to manually setup the gateways,routes, and DNS properly. I am doing more reading and trial and error with IP addressing.

    Any thoughts, Regit?

  14. Hi Regit, it is possible to use several instances of af-packet? that is: if I have several interfaces on my machine, it is posible to create in suricata.yaml: afpacket_instance1(eth0&eth1), afpacket_instance2(eth0&eth2), afpacket_instance3(eth0&eth3) and so on… How do you see it?

  15. Hello Mark. Yes this is possible. Just look at the suricata.yaml configuration file. It is already specified in the af-packet section.

    Once you have wrote your config file, you can start ‘suricata –af-packet’ and it will liste on all interfaces.

  16. Hi Regit, this mode look sound interesting. Did you has some test results when Suricata work as IPS in this mode?

  17. Hello Huy, some users are using it and it seems to perform well. It is said has been faster than NFQ mode.

  18. Chris, Joseph, and Regit,

    Did you ever solve the issue with the wDrop logging to fast.log but not actually dropping the packets? I’m seeing the same thing and I do have inline:yes

  19. Hi Regit,
    I run suricata in af-packet mode by command suricata -D -c suricata.yaml -i br0.
    With br0 is bridged of eth1&eth2.

    My drop rules is not performed.
    Reject rules is not performed too .

    This is rule:
    reject ip any any -> 8.8.8.8 any (msg:”Block ping DNS google”; sid:10000002; rev:1;)

    and

    drop ip any any -> 8.8.8.8 any (msg:”Block ping DNS google”; sid:10000002; rev:1;)

    Could u pls confirm with af-packet mode, Can I do drop/reject ?

  20. Hi all,

    I’m confused on the following statements about in IPS/inline mode with a host with 2 nics:
    “AF_PACKET IPS mode is completely separated from firewall stack.It establishes a software bridge between two interfaces by copying packet from one interface to the other (and reverse).”
    “Suricata will be in charge of transferring packets from one interface to the other.”

    What are then the basic network interface setting requirements related to routing on the OS level ? ip forwarding ? static routing tables ? default gateway ?

  21. I must be doing something wrong here, I have a VM with two nics attached. One is enp0s3 and the other enp0s8. I am running Suricata version 4.0.5.

    I have the following in my yaml in the af-packet section.

    af-packet:
    – interface: enp0s3
    threads: 1
    defrag: yes
    cluster-type: cluster_flow
    cluster-id: 98
    copy-mode: ips
    copy-iface: enp0s8
    buffer-size: 64535
    use-mmap: yes
    – interface: enp0s8
    threads: 1
    cluster-id: 97
    defrag: yes
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: enp0s3
    buffer-size: 64535
    use-mmap: yes

    I also have the following in the stream section

    stream:
    memcap: 32mb
    checksum-validation: yes
    inline: yes
    reassembly:
    memcap: 64mb
    depth: 1mb
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560

    I have the two network interfaces setup with static IPs

    $ ifconfig
    enp0s3 Link encap:Ethernet HWaddr 08:00:27:25:ac:2b
    inet addr:10.0.19.16 Bcast:10.0.19.255 Mask:255.255.255.0
    inet6 addr: fe80::b78f:c69a:b566:cb37/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:4131515 errors:0 dropped:0 overruns:0 frame:0
    TX packets:2690412 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:630894339 (630.8 MB) TX bytes:269331975 (269.3 MB)

    enp0s8 Link encap:Ethernet HWaddr 08:00:27:41:5b:de
    inet addr:10.0.19.17 Bcast:10.0.19.255 Mask:255.255.255.0
    inet6 addr: fe80::95a8:d0e9:d138:b746/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:2969442 errors:0 dropped:0 overruns:0 frame:0
    TX packets:3797496 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:626044650 (626.0 MB) TX bytes:272136409 (272.1 MB)

    My routing table only has routes for the enp0s3 nic

    $ route
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    default 10.0.19.1 0.0.0.0 UG 0 0 0 enp0s3
    10.0.19.1 * 255.255.255.255 UH 0 0 0 enp0s3

    So In my head, I’m thinking that I’d like to have anything routed from enp0s8 go be copied by suricata to enp0s3 only if it doesn’t have a drop rule. However, this doesn’t seem to be possible to create a rule to route from enp0s8 to enp0s3 in that way. I just get the error “Network is unreachable”.

    I’m using the command

    route add default gw 10.0.19.1 enp0s8

    Anyone know what I’m doing wrong?

  22. Hi,
    I am running suricata af_packet in IPS mode on marvell board, packets are not copying from one interface to another. When i stop suricata, I can see No. of packets count received on one interface connected to WAN, but packets are not copying on other interface given in af_packet setting.

Leave a Reply

Your email address will not be published. Required fields are marked *