As the founder of Catena Cyber and a member of the Suricata development team, Philippe Antoine has been fuzzing Suricata and other projects in OSS-Fuzz for some time. At FuzzCon Europe 2021, he demonstrated how he managed to find 49 unique bugs within the intrusion detection system in the past year.
Philippe Antoine - One Year of Fuzzing and Fixing Suricata
What Is Suricata?
Suricata is the leading independent open-source threat detection engine, combining intrusion detection (IDS), intrusion prevention (IPS), network security monitoring (NSM) and PCAP processing. It is written in C and Rust, and currently in the process of moving its C codebase to Rust for security.
In addition to its configuration (mostly a YAML file), Suricata’s inputs are rules written in a custom language (without a defined grammar yet) and network traffic, where an attacker is expected to control parts of it.
Results of More Than a Year of Fuzzing
Since the release of Suricata version 6.0.0, 49 unique bugs were found by OSS-Fuzz. 41 of them were only reported by OSS-Fuzz, meaning that 8 of the bug findings were also reported by other tools. 40 out of these 49 bugs are triggerable by network traffic, the 9 other bugs are located in the rules parsing.
The source for this data is public:
- OSS-Fuzz bug tracker: https://bugs.chromium.org/p/oss-fuzz/issues/list?q=label%3AProj-suricata
- Suricata bug tracker: https://redmine.openinfosecfoundation.org/
In a manual review, duplicates and false positives (bugs in the fuzz target but not in Suricata itself) were removed. The complete results are also available publicly.
This data is not exhaustive, as it only concerns bugs that reach the git master branch, and not those found by fuzzing during continuous integration. Throughout the project, bugs were found steadily:
Most bugs discovered in the code already existed before the update to version 6.0.0. Lately, 4 bugs were found in the git master branch, before they could make it into any release.
Only 6 of these bugs seem to induce a risk of remote code execution: 3 use-after-free and 3 buffer overflows. As Suricata is a cybersecurity software, a denial of service, such as one created by a null dereference or a rust panic is still a big issue. Partial denial of services, such as those created by memory leaks, quadratic complexity leading to timeouts, or evasions detected by failed assertions, are also a big threat as they are stealthier than a plain crash.
To complete this analysis, I also looked into the bugs that were missed by fuzzing. There were 34 redmine tickets closed for version 6.0.3, with 32 unique relevant bugs. 10 of them came from fuzzing, while the remaining 22 were missed.
The source for this analysis can be found here:
https://redmine.openinfosecfoundation.org/versions/167
https://catenacyber.fr/suricata-bugs.csv
The reasons for fuzzers missing these bugs are:
- Mainly missing a ground truth for (12 semantic bugs)
- Fuzzing only the latest master branch, and having one bug only in the version 6 branch
- Not using certain configuration options, or having bugs in code that is unreachable by the present harnesses (4 bugs)
- Unknown (5 bugs): Perhaps these bugs were caused by a lack of time waiting for blockers to be fixed
Improvements to OSS-Fuzz in the Last Year
To achieve these results in Suricata, some continuous improvements were brought to OSS-Fuzz.
Firstly, getting a coverage report for both C and Rust took some effort:
https://github.com/google/oss-fuzz/pull/4697
https://github.com/google/oss-fuzz/pull/5352
https://github.com/google/oss-fuzz/pull/6517
But this effort was not in vain, as the first coverage report showed an unexpectedly low coverage for rust (but not zero as bugs had already been found).
A closer analysis showed that the compilation flags were not properly set for rust fuzzing. Thus, by the end of the month, the coverage had increased from 21% to 71%, finding new bugs along the way. 
Check out the updated coverage report ( November 3rd, 2021)
Inspecting the coverage report, another improvement was to copy the same fuzz target multiple times fuzz_applayerparserparse, which uses its first byte of input to determine an app-layer protocol (such as HTTP, SMB, DNS, etc…) to parse on the rest of the input. This proved to be a successful change as the bugs found by these targets show:
- 1 fuzz_applayerparserparse_dcerpc
- 1 fuzz_applayerparserparse_dnp3
- 1 fuzz_applayerparserparse_ftp
- 2 fuzz_applayerparserparse_http2
- 1 fuzz_applayerparserparse_ike
- 1 fuzz_applayerparserparse_mqtt
- 1 fuzz_applayerparserparse_smtp
- 0 fuzz_applayerparserparse
Another change was to introduce a structure-aware fuzz target. Pcap is the de facto standard for network traffic representation. Since pcap is a complex format in itself and we wanted to fuzz Suricata instead of libpcap, we created a fuzzing-friendly libpcap-like library: https://github.com/catenacyber/fuzzpcap.
This allowed us to compute a list of packets for one single TCP stream with the sequences and acks numbers. With this in place, we do no longer fuzz the TCP reassembly engine, but rather the deep core of Suricata. While the new structure-aware target found 9 new bugs, the old target still found 1 bug since the introduction of the structure-aware target.
Further Improvements Are Coming
Despite all the enhancements, there is still room for improvement. As shown by a POC for getting a performance report with a flamegraph, we spend too much time (17%) in JsonAnomalyLogger for what it is worth.
Bringing positive contributions to Suricata encouraged all involved developers to use more assertions, to ensure that fuzzers can’t craft impossible inputs.
Improving Continuous Fuzzing Takes Good Reports
Coverage and performance reports are the input data that enabled us to conduct such an analysis. Additionally, asking oneself why a specific bug was not found by the fuzzers at first, but then reported otherwise, is also a good way to come up with further improvements.
Recap FuzzCon Europe
This blog is a summary of a talk Philippe Antoine held at FuzzCon Europe 2021. You can check out the FuzzCon Europe recap page to watch all recorded talks and live coding sessions of the event.
.webp?width=570&name=Get%20the%20Recordings%20(1).webp)