Ulogd on To Linux and beyond !

Suricata and Ulogd meet Logstash and Splunk

Fri, 07 Mar 2014 23:19:37 +0000

Some progress on the JSON side

Suricata 2.0-rc2 is out and it brings some progress on the JSON side. The logging of SSH protocol has been added:

and the format of timestamp has been updated to be ISO 8601 compliant and it is now named timestamp instead of time.

Nftables and the Netfilter logging framework

Mon, 24 Feb 2014 22:17:22 +0000

Nftables logging

If nftables is bringing a lot of changes on user side, this is also true in the logging area.
There is now only one single keyword for logging: log and this target is using the Netfilter logging framework.
A corollary of that is that why you may not see any log messages even if a rule with log is matching because the Netfilter logging framework has to be configured.

Using ulogd and JSON output

Sun, 02 Feb 2014 16:39:34 +0000

Ulogd and JSON output

In February 2014, I’ve commited a new output plugin to ulogd, the userspace logging daemon for Netfilter. This is a JSON output plugin which output logs into a file in JSON format. The interest of the JSON format is that it is easily parsed by software just as logstash. And once data are understood by logstash, you can get some nice and useful dashboard in Kibana:

Investigation on an attack tool used in China

Sun, 02 Feb 2014 15:28:32 +0000

Log analysis experiment

I’ve been playing lately with logstash using data from the ulogd JSON output plugin and the Suricata full JSON output as well as standard system logs.

Ulogd is getting Netfilter firewall logs from Linux kernel and is writing them in JSON format. Suricata is doing the same with alert and other traces. Logstash is getting both log as well as sytem log. This allows to create some dashboard with information coming from multiple sources. If you want to know how to configure ulogd for JSON output check this post. For suricata, you can have a look at this one.

Some ulogd db improvements

Tue, 21 May 2013 21:42:57 +0000

Some new features

I’ve just pushed to ulogd tree a series of patches. They bring two major improvements to database handling:

Backlog system: temporary store SQL query in memory if database is down.
Ring buffer system: a special mode with a thread to read data from kernel and a thread to do the SQL query.

The first mode is attended for preventing data loss when database is temporary down. The second one is an attempt to improve performance and the resistance to netlink buffer overrun problem.
The modification has been done in the database abstraction layer and it is thus available in MySQL, PostgreSQL and DBI.

Visualize Netfilter accounting in Graphite

Sat, 22 Dec 2012 11:27:09 +0000

Ulogd Graphite output plugin

I’m committed a new output plugin for ulogd. The idea is to send NFACCT accounting data to a graphite server to be able to display the received data. Graphite is a web application which provide real-time visualization and storage of numeric time-series data.

Once data are sent to the graphite server, it is possible to use the web interface to setup different dashboard and graphs (including combination and mathematical operation):