New features in ulogd 2.0.3 release
Database framework update
ulogd 2.0.3 implements two new optional modes for database connections:
- backlog system to avoid event loss in case of database downtime
- running mode where acquisition is made in one thread and queries to databases are made in separate threads to reduce latency in the treatment of kernel messages
These two modes are described below.
Postgresql update
Postgresql output plugin was only offering a small subset of Postgresql connection-related options.
It is now possible to use the connstring to use all possible parameters of libpq param keywords. If set, this variable has precedence on other variables.
One interest of connstring is to be able to use a SSL-encrypted connection to the database by using the sslmode keyword:
connstring="host=localhost port=4321 dbname=nulog user=nupik password=changeme sslmode=verify-full sslcert=/etc/ssl/pgsql-cert.pem sslkey=/etc/ssl/pgsql-key.pem sslrootcert==/etc/ssl/pgsql-rootcert.pem"
Event loss prevention
ulogd 2.0.3 implements a backlog system for all database output plugins using the abstraction framework for database connection. At the writing of this article, this is MySQL, PostgreSQL and DBI. Memory will be dedicated to store the queries that can not be run because of an unavailability of the database. Once the database is back, the queries are played in order.
To activate this mode, you need to set the backlog_memcap value in the database definition.
[mysql1] db="nulog" ... procedure="INSERT_PACKET_FULL" backlog_memcap=1000000 backlog_oneshot_requests=10
Set backlog_memcap to the size of memory that will be allocated to store events in memory if data is temporary down. The variable backlog_oneshot_requests is storing the number of queries to process at once before reading a kernel message.
Multithreaded database output
If the ring buffer mode is active, a thread will be created for each stack involving the configured database. It will connect to the database and execute the queries.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side. With this new mode, the time to process kernel message is equal to the time of the formatting of the query.
To activate this mode, you need to set ring_buffer_size to a value superior to 1.
The value stores the number of SQL requests to keep in the ring buffer.
[pgsql1] db="nulog" ... procedure="INSERT_PACKET_FULL" ring_buffer_size=1000
The ring_buffer_size has precedence on the backlog_memcap value. And backlog will be disabled if the ring buffer is active as ring buffer also provide packet loss prevention. ring_buffer_size is the maximum number of queries to keep in memory.