Menu

One Year of Fuzzing and Fixing Suricata

November 18 2021

As the founder of Catena Cyber and a member of the Suricata development team, Philippe Antoine has been fuzzing Suricata and other projects in OSS-Fuzz for some time. At FuzzCon Europe 2021, he demonstrated how he managed to find 49 unique bugs within the intrusion detection system in the past year.

Philippe Antoine - One Year of Fuzzing and Fixing Suricata

What Is Suricata?

Suricata is the leading independent open-source threat detection engine, combining intrusion detection (IDS), intrusion prevention (IPS), network security monitoring (NSM) and PCAP processing. It is written in C and Rust, and currently in the process of moving its C codebase to Rust for security. 

In addition to its configuration (mostly a YAML file), Suricata’s inputs are rules written in a custom language (without a defined grammar yet) and network traffic, where an attacker is expected to control parts of it. 

Network Traffic - Flowchart

Results of More Than a Year of Fuzzing

Since the release of Suricata version 6.0.0, 49 unique bugs were found by OSS-Fuzz. 41 of them were only reported by OSS-Fuzz, meaning that 8 of the bug findings were also reported by other tools. 40 out of these 49 bugs are triggerable by network traffic, the 9 other bugs are located in the rules parsing.  

The source for this data is public: 

In a manual review, duplicates and false positives (bugs in the fuzz target but not in Suricata itself) were removed. The complete results are also available publicly 

This data is not exhaustive, as it only concerns bugs that reach the git master branch, and not those found by fuzzing during continuous integration. Throughout the project, bugs were found steadily:

Found bugs - Barchart

Most bugs discovered in the code already existed before the update to version 6.0.0. Lately, 4 bugs were found in the git master branch, before they could make it into any release. 

Pre-6 Version - Pie chart

Only 6 of these bugs seem to induce a risk of remote code execution: 3 use-after-free and 3 buffer overflows. As Suricata is a cybersecurity software, a denial of service, such as one created by a null dereference or a rust panic is still a big issue. Partial denial of services, such as those created by memory leaks, quadratic complexity leading to timeouts, or evasions detected by failed assertions, are also a big threat as they are stealthier than a plain crash. 

Overview - Pie chart

To complete this analysis, I also looked into the bugs that were missed by fuzzing. There were 34 redmine tickets closed for version 6.0.3, with 32 unique relevant bugs. 10 of them came from fuzzing, while the remaining 22 were missed. 

The source for this analysis can be found here: 
https://redmine.openinfosecfoundation.org/versions/167 
https://catenacyber.fr/suricata-bugs.csv

The reasons for fuzzers missing these bugs are:

  • Mainly missing a ground truth for (12 semantic bugs) 
  • Fuzzing only the latest master branch, and having one bug only in the version 6 branch
  • Not using certain configuration options, or having bugs in code that is unreachable by the present harnesses (4 bugs) 
  • Unknown (5 bugs): Perhaps these bugs were caused by a lack of time waiting for blockers to be fixed 
Missed Bugs - Pie chart

Improvements to OSS-Fuzz in the Last Year

To achieve these results in Suricata, some continuous improvements were brought to OSS-Fuzz. 

Firstly, getting a coverage report for both C and Rust took some effort: 
https://github.com/google/oss-fuzz/pull/4697 
https://github.com/google/oss-fuzz/pull/5352 
https://github.com/google/oss-fuzz/pull/6517 

But this effort was not in vain, as the first coverage report showed an unexpectedly low coverage for rust (but not zero as bugs had already been found). 

Coverage Report 1

A closer analysis showed that the compilation flags were not properly set for rust fuzzing. Thus, by the end of the month, the coverage had increased from 21% to 71%, finding new bugs along the way. Coverage Report 2

Check out the updated coverage report ( November 3rd, 2021)

Inspecting the coverage report, another improvement was to copy the same fuzz target multiple times fuzz_applayerparserparse, which uses its first byte of input to determine an app-layer protocol (such as HTTP, SMB, DNS, etc…) to parse on the rest of the input. This proved to be a successful change as the bugs found by these targets show:

  • 1 fuzz_applayerparserparse_dcerpc
  • 1 fuzz_applayerparserparse_dnp3
  • 1 fuzz_applayerparserparse_ftp
  • 2 fuzz_applayerparserparse_http2
  • 1 fuzz_applayerparserparse_ike
  • 1 fuzz_applayerparserparse_mqtt
  • 1 fuzz_applayerparserparse_smtp
  • 0 fuzz_applayerparserparse 

Another change was to introduce a structure-aware fuzz target. Pcap is the de facto standard for network traffic representation. Since pcap is a complex format in itself and we wanted to fuzz Suricata instead of libpcap, we created a fuzzing-friendly libpcap-like library: https://github.com/catenacyber/fuzzpcap.

This allowed us to compute a list of packets for one single TCP stream with the sequences and acks numbers. With this in place, we do no longer fuzz the TCP reassembly engine, but rather the deep core of Suricata. While the new structure-aware target found 9 new bugs, the old target still found 1 bug since the introduction of the structure-aware target. 

Further Improvements Are Coming

Despite all the enhancements, there is still room for improvement. As shown by a POC for getting a performance report with a flamegraph, we spend too much time (17%) in JsonAnomalyLogger for what it is worth. 

Flame Graph

Bringing positive contributions to Suricata encouraged all involved developers to use more assertions, to ensure that fuzzers can’t craft impossible inputs. 

Improving Continuous Fuzzing Takes Good Reports

Coverage and performance reports are the input data that enabled us to conduct such an analysis. Additionally, asking oneself why a specific bug was not found by the fuzzers at first, but then reported otherwise, is also a good way to come up with further improvements. 


Recap FuzzCon Europe 2021

This blog is a summary of a talk Philippe Antoine held at FuzzCon Europe 2021. You can check out the FuzzCon Europe 2021 recap page to watch all recorded talks and live coding sessions of the event.Recap FuzzCon Europe 2021Get the Recordings

 

Recent Posts

One Year of Fuzzing and Fixing Suricata

Autofuzz: Fuzzing Without Writing Fuzz Targets or Harnesses

Fuzzing 101 – The Basics (FAQ)

19 Bugs in Jsoup Found With Jazzer

Share Article

Subscribe to updates