Fabian Meumertzheim is a Senior Software Engineer at Code Intelligence and one of the leading engineers behind Jazzer, Code Intelligence’s open-source fuzzer for JVM-based languages. In this article, he gives a brief introduction on how to fuzz with Jazzer.
Jazzer is an open-source fuzzing engine for the Java Virtual Machine (JVM). With Jazzer, developers can increase their test coverage to find edge cases and avoid software bugs more effectively.
We built Jazzer based on popular and proven tools, including the libFuzzer fuzzing engine and JaCoCo for coverage instrumentation. Like most modern fuzzers, Jazzer is a coverage-guided and in-process fuzzer, which means that it runs Java targets in its own JVM instance while monitoring the reached code paths. This allows developers to test their software at high speed. Since Jazzer operates directly on the JVM bytecode, this is possible even without access to source code.
Jazzer has all the advanced libFuzzer features and ports them to Java while also adding some additional Java smarts, like a “keep going” mode. This mode allows developers to quickly skip uninteresting exceptions in favor of more interesting bugs and vulnerabilities.
Tried and Tested libFuzzer Features:
Additional Java Features:
With Jazzer, we trigger a lot of bugs. But as a Java developer, you might want to debug the code without all the fuzzer "overhead". For this, we create simple java classes with a main function triggering the malicious inputs, so-called reproducers.
Method hooking is an advanced feature that can be used to write bug detectors and sanitizers (for more information on Method hooking, have a look at our GitHub documentation.)
Below, you can see a small but challenging example application. The application consumes a String and two longs. First, it encodes the String using Base64 and checks whether it equals a predefined String. (SmF6emVy is the Base64 encoding of “Jazzer”). Only if the Base64 encoding matches does the program continue. Next, the application uses a simple XOR encryption with a static key on the two longs and again compares them to predefined values. If both match, the program continues and reaches the planted bug. While this is a simple example for humans to solve, fuzzers can easily struggle with this kind of code since very specific “magic” values need to be found to pass the checks. In this case, the fuzzer also needs to “break” the XOR encryption.
Simple application. See full gist.
Jazzer, however, can solve this problem with very little effort. Below you see the Jazzer fuzz target, which is basically just a static fuzzerTestOneInput function. The function takes a FuzzedDataProvider, a helper object for splitting the raw fuzzer input into useful primitive types, and gets the String and two longs required by the sample application. Then all it does is call the application with that data. It is as simple as that.
Simple fuzz target. See full gist.
If you run Jazzer, it will only take about five seconds to crash this target. After about 2.5 million runs with over 600 000 runs per second, Jazzer will have found the “magic” values that get it past the Base64 encoding and XOR encryption and trigger the bug. The reason Jazzer can get past these hurdles so quickly is that like libFuzzer, Jazzer uses value profiling to guide the fuzzer past these kinds of checks much more efficiently than simply hoping to stumble on the exact value by chance. For more on this see https://llvm.org/docs/LibFuzzer.html#value-profile and https://github.com/CodeIntelligenceTesting/jazzer/#value-profile.
Jazzer output. See full gist
Among developers, it is a common misconception, that fuzz testing is only suitable for detecting memory corruption bugs. And since Java is a memory safe language, there is no need to fuzz Java.
However, fuzzers can find many more bugs than only memory corruptions. It’s just that languages like C and C++ have so many memory bugs with high impact that they get all the attention. Indeed, fuzzers can find any type of bug you care to specify. Simple yet common examples are uncaught exceptions and memory leaks that crash services or applications. Infinite loops that can be used in DoS attacks can also be found. But more interestingly, anything that can be expressed as an assertion can be found quite simply, e.g. multiple implementations or libraries that should produce the same output for a given input but diverge in edge cases.
It is also possible to find SQL injections or even remote code execution bugs. So in our view, there is no question that memory-safe languages need to be fuzzed since all languages are susceptible to functional bugs as well as severe security vulnerabilities.
Let's have a look at a real bug we found with Jazzer. In this particular use case, we will talk about the JSON sanitizer, which is a popular Java library developed at Google and maintained by the OWASP Foundation. You can feed the JSON-sanitizer arbitrary bytes and strings while it ensures that the output is always valid JSON. The JSON Sanitizer guarantees that the output will never contain certain substrings that, due to the nature of how HTML is parsed, might mess up your scripts or even cause XSS. Thus, after running untrusted input through the sanitizer, you should be able to safely embed it into a script block in your HTML, like in the orange box below.
JSON-sanitizer: example of an embedded script.
We set this up with a property-based fuzzing approach in Jazzer. We consume a string and pass it through the sanitizer. Then, we assert that the “safe” JSON that comes out of the sanitizer does not contain the specified substring. Because if it does, we would be vulnerable to cross-site scripting.
Property-based fuzz target. See full gist.
XXS-Bug in JSON Sanitizer (CVE-2021-23899).
Long story short: Fuzzing allowed us to find a high severity security vulnerability in this popular and well-tested Java project with very little effort.
Update: Google integrated Jazzer into OSS-Fuzz. Now open-source projects can leverage Google’s infrastructure and computational power to secure their Java libraries. Read the full release note on the Google Security Blog.
Since Jazzer's release in mid-February 2021, we have already found over 100 bugs in more than 20 open-source projects that we fuzzed. 8 out of the 50 bugs were security issues, 5 of which were found in stable releases and thus assigned CVEs. (See full trophy list).
I hope this post gave you a good first impression of how easy it is to find bugs in Java applications with Jazzer. If you want to learn more about Jazzer, I also recommend the article on “Engineering Jazzer.” Or just check out Jazzer on GitHub.
If you find a bug with it, I would be happy to hear from you (once it’s public) on Twitter (@fhenneke).
For more exciting talks on fuzz testing and web security, visit fuzzcon.eu.
Fabian Meumertzheim is a Senior Software Engineer at Code Intelligence. He maintains and contributed to multiple open-source projects, such as Chromium, Jazzer, systemd, and Android Password Store, all with the aim of making security unobtrusive and ubiquitous.