OISF brainstorming: planning phase 3 (take 3)

GEO IP

Idea is to add a keyword that would be used to interact with GEOIP database (free at least) and be able to use it to detect things like control canal. For example, an IRC server in an non common country is certainly a control canal.

Live ruleset swap

A must have! This is vital for critical environnement. This is very costly in memory and this should be an option to avoid exploding low memory boxes.

Qosmos integration / API for data exchange

Bringing protocol analysis is an interesting point as it will help to increase performance and accuracy of the engine. Knowing the protocol permit to only run protocol related rules to flow of that protocol. And this avoid to have false detection by running the rules on bad protocol. OpenDPI technology and Qosmos technology integration is discussed. A common API is needed to be able to use both systems.

Global shared flowvars

Global flow var will permit to change the way we build rules. Not being constrained anymore to stream variable will increase the power of rules.

Host/app/OS table import

Idea is to load host type from file to be able to tune the host setting precisely.

IPFIX support

IPFIX support as entry or output could bring some advantages.

Conclusion

Matt Jonkman and Victor Julien will now summarize the input and publish on OISF website the planned features for phase 3 based on discussion about priority of the tasks that have been held.

OISF brainstorming: planning phase 3 (take 2)

DNS fast flux/anomaly detection

The idea is to detect malware and other things by collecting the DNS request and their answer and detecting anomaly. For example, if an host is making a lot of request to a domain.

First part of the job on Suricata is to log all requests and their answer. Then analysis can occurs in the database.

File extraction

This is a work under progress linked with a third party contract. It permit to store exchanged files on disk for some application level protocol. It is possible to say: “store the file, if the content type is different from the extension”. File extraction works currently on HTTP. It focus on POST request to detect uploaded file.

Time-based counters

This aims at detection of regular behaviours which are often linked with command control connection. For example, triggering an alert if a specific DNS request is done every five minutes.

HTTP keyword improvement

HTTP keywords improvement are discussed, some specific keywords could be added to avoid the cost of pcre. A two parameters header match is suggested to support all possible keyword.

Output

Discussion is about logging the normalised application level content. Currently, only the packet triggering the lart is loggued and thus the information about why suricata has logged is lost. This is thus interesting to log the reconstructed application level message to permit the analyst to analyse the reason of the alert.

Regarding the output module, it could be interesting to adds support for CEE logging which would be able to support the resulting composite alert. For performance reason, a barnyard output of this composite alert is interesting. It may be needed to suggest some change to support this application level alert.

Oisf brainstorming: planning phase 3 (take 1)

Performance improvement

As shown by Victor’s latest work on performance counters, there is a lot of work that can be done to improve performance. They are currently good but there is place for improvement. Proposal to provide off-loading or clustering is done. This is heavily discussed but as pointed out by Victor, it will be more interesting to do this in the next phase. Phase 3 should focus in improvement of current code. This will permit to use the upcoming Suricata killing features like global flow variable.

SSL preprocessor

Following the recent certicate authority attacks, a SSL preprocessor which is able to detect blacklist certificate and other things will be really interesting. It could also detect certificate property change whan connection to host or similar things.

Decryption is not seen by the participants of the webex has important. Decryption would be performance killer without accelerator. Thus a limitation to certificate analysis is at first an interesting target

IP and DNS reputation

The idea is too exchange IP reputation between sensors to protect themself against offensive hosts as soon as it has been identified on one sensor. This requires exchange between nodes and this is already done by other projects. Doing it alone in Suricata would be to big and thinking about adding a way to interact easily with this project could be a first step.

Matt Jonkman: development avancement

Phase 2 development is almost over now. Among the completed major features:

  • Multithread
  • protocol discovery
  • smb logging
  • HTTP logging
  • flowvars

One of the advantage of Suricata over Snort is protocol discovery combined to HTTP parsing by libhtp. It provides a huge improvement over Snort as a lot of bad flow are using HTTP on non standard ports.

Victor Julien: Development status

Work has started in september 2007. The work depends on some externel library like multithread of input handling library. The main external depedency is libhtp which is initally developped by Ivan Ristic.

The development is managed in a single git repository. Victor is the only one with commit right. The review are done by Victor and cross review are made by developpers.

Work unit for developers are tasks which are written by Victor and describe a specific task to do. This task are mainly done by OISF funded developers. Some simpler task are let to the comunity and everyone can help with this.

To offload Victor’s load, subsystem mainteners are nominated:

  • Eric Leblond: packet acquisition
  • Anoop Saldanha: detection part

They will have freedom on the way to improve the subsystem they are in charge.

The development is currently done with two branches (1.0 which is bug fix only and master). Victor is currently not happy with this and would like to switch to time-based release. This is discussed as this can be difficult for company to deal with frequent updates. A funding of maintenance by companies could help to keep the current working system.

Peter Manev is in charge of the QA. There is a lot of work to do in this area. Unit test is currently good but there is a lot of work to do to improve detection of regression.

Performance has been improved in 1.1 with a focus on efficient algorithm.

CUDA support needs help. The performance is still lower with than without and thus this really need developer power!

Performance profiling have been added recently by Victor and it shows clearly that work is needed on this.

To sum up, the two main areas where help will be most than welcome are QA and performance profiling.

Matt Jonkman: introduction speech

Matt presents the goal of the OISF brainstorming session:

  • Make a status of the foundation
  • Grabbing new ideas

The session will be interactive and anybody is invited to participate through physical intendance or webex.

The foundation is non-profitable and aim at building a powerful engine for us all. OISF is member og the HOST program and happily supported by some industrials.

Foundation business

Matt fills he can not give enough times to the foundation due to his work at EmergingThreat and propose to hire a General Manager that would take care of finding the funding and administrative part.

The funding renewal was smallest than planned and the work on getting funding must be increased. Situation is not bad but improvement could be interesting.

Potential major projects

Universal Rules Language

This is a highly difficult problem. It has been wanted by major actors for years but no successfull and clean approach has been found.

Commercial support

A lot of industrials want to have access to commercial support to integrate Suricata in their product. Developers can not do the job but they can form some support people to have them become experts.

OpenDPI/Qosmos integration

The idea is to add support for multiple protocols to be able to provide high level rules for a lot more of protocols.

Scada

Digital bound give a copyright usage to the foundation to integrade a preprocessor into Suricata.

Snortsam integration

Snortsam is now a plugin to barnyard and this made it more easy to use.