Some progress on the JSON side Suricata 2.0-rc2 is out and it brings some progress on the JSON side. The logging of SSH protocol has been added: and the format of timestamp has been updated to be ISO 8601 compliant and it is now named timestamp instead of time. ...
Nftables logging If nftables is bringing a lot of changes on user side, this is also true in the logging area. There is now only one single keyword for logging: log and this target is using the Netfilter logging framework. A corollary of that is that why you may not see any log messages even if a rule with log is matching because the Netfilter logging framework has to be configured. ...
Ulogd and JSON output In February 2014, I’ve commited a new output plugin to ulogd, the userspace logging daemon for Netfilter. This is a JSON output plugin which output logs into a file in JSON format. The interest of the JSON format is that it is easily parsed by software just as logstash. And once data are understood by logstash, you can get some nice and useful dashboard in Kibana: ...
Log analysis experiment I’ve been playing lately with logstash using data from the ulogd JSON output plugin and the Suricata full JSON output as well as standard system logs. Ulogd is getting Netfilter firewall logs from Linux kernel and is writing them in JSON format. Suricata is doing the same with alert and other traces. Logstash is getting both log as well as sytem log. This allows to create some dashboard with information coming from multiple sources. If you want to know how to configure ulogd for JSON output check this post. For suricata, you can have a look at this one. ...
Some new features I’ve just pushed to ulogd tree a series of patches. They bring two major improvements to database handling: Backlog system: temporary store SQL query in memory if database is down. Ring buffer system: a special mode with a thread to read data from kernel and a thread to do the SQL query. The first mode is attended for preventing data loss when database is temporary down. The second one is an attempt to improve performance and the resistance to netlink buffer overrun problem. The modification has been done in the database abstraction layer and it is thus available in MySQL, PostgreSQL and DBI. ...
Ulogd Graphite output plugin I’m committed a new output plugin for ulogd. The idea is to send NFACCT accounting data to a graphite server to be able to display the received data. Graphite is a web application which provide real-time visualization and storage of numeric time-series data. Once data are sent to the graphite server, it is possible to use the web interface to setup different dashboard and graphs (including combination and mathematical operation): ...