Some new features
I’ve just pushed to ulogd tree a series of patches. They bring two major improvements to database handling:
- Backlog system: temporary store SQL query in memory if database is down.
- Ring buffer system: a special mode with a thread to read data from kernel and a thread to do the SQL query.
The first mode is attended for preventing data loss when database is temporary down. The second one is an attempt to improve performance and the resistance to netlink buffer overrun problem.
The modification has been done in the database abstraction layer and it is thus available in MySQL, PostgreSQL and DBI.
Backlog system
To activate this mode, you need to set the backlog_memcap value in the database definition.
[mysql1] db="nulog" ... procedure="INSERT_PACKET_FULL" backlog_memcap=1000000 backlog_oneshot_requests=10
The backlog system will prevent data loss by storing queries in memory instead of executing them. The waiting queries will be run in order when the connection to the database is restored.
Ring buffer setup
To activate this mode, you need to set ring_buffer_size to a value superior to 1.
The value stores the number of SQL requests to keep in the ring buffer.
[pgsql1] db="nulog" ... procedure="INSERT_PACKET_FULL" ring_buffer_size=1000
The ring_buffer_size has precedence on the backlog_memcap value. And backlog will be disabled if the ring buffer is active.
If the ring buffer mode is active, a thread will be created for each stack involving the configured database. It will connect to the database and execute the queries.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.
Conclusion
Feel free to test it and don’t hesitate to provide some feedback!