Skip to content
Close
Login
Login

What Is Fuzz Testing?

Fuzzing is a dynamic application security testing method used for finding functional bugs
and security issues in software.

Trusted by

What to Expect on this Page

 

What Are the Benefits of Fuzz Testing?

Fuzz testing enables developers to ship secure software fast, by detecting security and stability issues in the early stages of software development.

1. Run Security Tests On the Source Code

During a fuzz test, a program gets executed with invalid, unexpected, or random inputs, with the aim to crash the application. Modern fuzzing solutions can analyze the structure of the code they are supposed to test. They can generate thousands of automated test cases per second, and mark each path the inputs take through the program. This way, a fuzzer gets detailed feedback about the code coverage, the inputs are reaching during the execution of the source code.

2. Each Finding Leads to More Findings

Once a fuzzer found an input that has caused a crash, it uses mutation algorithms to generate even more inputs that can reproduce the finding with a high probability. 

3. Fuzzing Protects Against Unexpected Edge Cases

Modern software fuzzers execute a program with invalid, unexpected, or random inputs. This way you can also cover unlikely or unexpected edge cases, that you would not cover with other testing approaches. 

4. Fuzzing Maximizes Code Coverage Without False Positives

Since fuzzers actually execute the software under test, they always provide inputs that you can use to reproduce the bug. This way, fuzzing enables you to reach up to 99% code coverage without any false positives. Fuzzing speeds up your development process and helps you to ship more reliable and secure software.

Fuzzing Explained in 120 Seconds

More and More Industry Standards Require Fuzz Testing

Due to increasing security regulations, more and more software companies have to run automated security tests before shipping their software. That's why many industries and ISO standards recommend integrating automated fuzz testing into the development process. Especially in industries, that already have advanced quality and security regulations. A good example is ISO/SAE 21434 and UNECE WP.29, which deal with the security of automotive software.

What Standards and ISO Norms Recommend Fuzzing?

ISO 26262
Road vehicles – Functional Safety

UNECE WP.29
United Nations World Forum for Harmonization of Vehicle Regulations

ISA/IEC 62443-4-1
Secure Product Development Lifecycle Requirements

ISO/SAE DIS 21434
Road Vehicles — Cybersecurity Engineering

UL2900-1 and UL2900-2-1
Healthcare and Wellness Systems - Software Cybersecurity for
Network-Connectable Products

ISO/IEC/IEEE 29119
Software and Systems Engineering - Software Testing

ISO/IEC 12207
Systems and Software Engineering – Software Life Cycle Processes

ISO 27001
Information Technology – Security Techniques – Information Security Management Systems

ISO 22301
Security and Resilience — Business Continuity Management Systems

IT-Grundschutz (Germany)
Based on ISO 27001

NIST SP 800-95
Web Services — standard for software testing (USA) and others

 

Microsoft and Google Are Driving Industry Research

Tech corporations like Microsoft and Google were early adopters of fuzzing technologies to test their own systems with it. Google released ClusterFuzz, the first cloud-based fuzzing infrastructure, in 2012 as part of a larger open source strategy. Since then, numerous open source tools (such as AFL, libFuzzer, HonggFuzz, and OSS-Fuzz) have also emerged, enabling more and more people to use fuzzing technologies. At this point, it was already becoming apparent that fuzzing was going to be a best practice in security testing. However, one major problem that was still looming at the time was the usability of fuzz testing. Open source tools such as AFL and libFuzzer were primarily developed by security experts for security experts and were intended for very specific fields of application. At first, normal developers could do little with them.

Best Practice: How Is Fuzzing Used in Practice?

Fuzzing is most effective when testing is done continuously. Therefore, it is a good idea to integrate fuzzing into CI/CD pipelines. This enables short feedback cycles and makes it possible for developers to quickly fix security vulnerabilities before they become a problem. But when implementing CI/CD fuzzing, developers should be careful not to block the pipeline for too long. This is because fuzzing can take a very long time in some circumstances. This problem can be solved by defining fixed time periods in which fuzzing should take place. For example, at night, between 10 pm and 6 am. Or, by performing regression tests on a regular basis. The fuzzer can also take over the corpus and crash inputs from past test runs, so that you don't have to start from scratch with every test.
When working together in larger development teams, it can also be useful to integrate fuzzing tools into code hosting systems like Gitlab or GitHub. This facilitates team communication and alerts developers if a problem is found during a pull request. Some commercial fuzzing solutions offer such integrations as well as helpful interfaces to issue-tracking systems like Jira and Jenkins.

When vulnerabilities in dependencies of modern Java applications are exploited, they can be used to maliciously spread data throughout multiple microservices rapidly. Since many microservices fully trust inputs from internal sources, attackers can often access large parts of the entire web backend through a small entry point. This was most recently shown by the RCE found in the popular Java library log4j, which affected thousands of Java applications around the world.

Being easier to maintain and more fail-safe, the complexity of modern microservice architectures leaves little room for error. Correspondingly, Java applications have to be tested thoroughly and with testing tools that can deal with their complexity.

Everything You Need to Know About Fuzzing at a Glance

Learn how you can apply fuzzing to build more stable and secure applications.

Download Fuzzing Fact Sheet

Fuzzing 101 - Fuzzing FAQs

New to fuzzing? As fuzzing is seeing increasing adoption in the dev community, we want to answer your questions about this topic as good as we can. In the below FAQ, we want to get you acquainted with fuzzing so you can create your own fuzz tests soon. Let's start with the basics. Feel free to skip ahead if you are already familiar with this.

1. What Is a Fuzzer? 

Fuzzing is a dynamic testing method used to identify bugs and vulnerabilities in software. It is mainly used for security and stability testing of the codebase. A fuzzer tests the software under test by feeding it with a series of inputs. In their most basic form, fuzzers generate these test inputs at random or based on a predefined set of values.

This bare-bones form of fuzzing is a black-box approach, that is often used by attackers, as it does not require access to the source code. Black-box fuzzers are easy to set up and inexpensive. However, they are not overly sophisticated, which has earned them the name dumb fuzzers in some circles. The more sophisticated counterpart to so-called dumb fuzzers are feedback-based fuzzers, also called smart fuzzers.

2. What Is Feedback-Based Fuzzing?

As the name implies, feedback-based fuzzers collect feedback about the code covered during the execution of inputs. Unlike testing tools that generate test inputs randomly, this information allows feedback-based fuzzers to constantly refine and improve test inputs and to monitor which parts of the program they reach. By leveraging code coverage, feedback-based fuzzers can traverse larger parts of the source code and trigger deeply hidden bugs and vulnerabilities. Since code coverage is one of the most important metrics in software security testing, feedback-based fuzzing tools can provide useful reports that aid in the interpretation of test findings.

3. Why Is Fuzzing (Especially) Useful for Security Testing?

There are some features that make software fuzzing enormously useful for security testing. Here is why: 

  • Fuzzing can be almost completely automated  
Fuzzing tools do not only find vulnerabilities; they also provide the inputs that trigger them 
  • Fuzzing detects bugs without false positives  

4. What Is a Fuzz Target?

Fuzz targets are small programs that test predefined API functions, similar to unit tests. However, the inputs are not provided by the developer, but produced by fuzz generators. The generators are responsible for creating random mutations of inputs that are sent to the software under test (SUT). The output of a fuzz generator (i.e., random inputs) is then sent to the SUT. The delivery mechanism processes inputs from the fuzz generator and feeds them to the SUT for execution.   

Finally, the monitoring system keeps track of how inputs are executed within the SUT and detects triggered bugs. This plays a critical part in the software fuzzing process, as it also influences what types of vulnerabilities can be discovered during fuzzing.  

Example of a Fuzz Target, for Java Applications

Click here to learn how to set up fuzz targets yourself.

public class ExampleValueProfileFuzzer {
// consume data from fuzzer and give it to function under test
public static void fuzzerTestOneInput(FuzzedDataProvider data) {
String str = data.consumeString(100); // get a string with max_len 100
long input1 = data.consumeLong();
long input2 = data.consumeLong();
// call function under test with fuzz data
ApplicationUnderTest.consume(str, input1, input2);
}

5. How Do I Set Up Continuous Fuzzing?

Continuous fuzzing enables dev team to automatically test their code within their favorite dev tools. The effort required to set up a continuous fuzzing cycle with open-source fuzzers is often underestimated. It requires a huge amount of manual configuration and maintenance, while usually not delivering the same results as an enterprise solution. With an enterprise solution, the set-up can be done much faster by simply following an installation manual. You can find out what such an enterprise solution looks like in our free demo app

6. How Can I Debug My Code After a Fuzz Test?

Modern fuzzing tools enable debugging with a few simple clicks. Since the code is executed during fuzz testing, the crashing input can easily be identified. As presented below, modern fuzzing platforms can immediately take users to the section of the code and set break-points in the stack trace, which enables effortless debugging.

Fuzzing 101- debug

7. What Are the Benefits of Fuzzing Compared to Other Testing Methods?

If you are looking for a way to secure your software, there are a variety of testing approaches, such as Static Applications Security Testing (SAST), Dynamic Application Security Testing (DAST), Interactive Application Security Testing (IAST), and Feedback-Based Application Security Testing (FAST). Each of these methods has its advantages and disadvantages. We have collected some of them in the matrix below.   

FuzzingIsTheMostScalableApproach1

Fuzz Testing compared to other Application Security Testing approaches, such as SAST, DAST and IAST. 

When vulnerabilities in dependencies of modern Java applications are exploited, they can be used to maliciously spread data throughout multiple microservices rapidly. Since many microservices fully trust inputs from internal sources, attackers can often access large parts of the entire web backend through a small entry point. This was most recently shown by the RCE found in the popular Java library log4j, which affected thousands of Java applications around the world.

While modern microservice architectures are easier to maintain and more fail-safe, their complexity leaves little room for error. Accordingly, Java applications have to be tested thoroughly and with testing tools that can manage their complexity.

8. What Are the Benefits of Fuzzing C/C++?

Fuzzing is tremendously effective at securing Memory Corruptions and other C/C++-typical vulnerabilities, which makes it highly popular among C/C++ developers. Google's open-source fuzzer OSS-Fuzz, was able to single-handedly uncover over 50 000 bugs in 300+ open-source applications, in just three years. The majority of these applications were written in C/C++. However, the great track record of C/C++-fuzzing, caused some people to falsely assume that finding memory corruptions in C/C++ is the only thing that fuzzing is good at. 

9. What Are the Benefits of Fuzzing Java Kotlin & Go?

Next to the memory-unsafe C/C++, fuzzing is nowadays used in a variety of memory-safe languages, such as Java, Kotlin, and Go. In these languages, fuzzing is highly effective at testing APIs and uncovering critical web vulnerabilities such as the OWASP top 10.  At Code Intelligence, we open-sourced our very own Java fuzzer Jazzer, which has since been integrated into Google's OSS-Fuzz.

A common misconception surrounding memory-safe languages is that they are inherently secure. Yes, memory-safe languages use runtime error prevention mechanisms, which basically make them immune to memory corruptions. However, there is a variety of other vulnerabilities that can occur in memory-safe languages such as Cross-Site-Scripting (XXS), Denial-of-Services (DoS) and Wrong Handled Exceptions, just to name a few.

How Fuzzing Prevented a Total Ethereum Shutdown

Fuzz testing finds bugs that other testing methods cannot detect. In November 2020 a serious DoS vulnerability was fixed in the source code of the Ethereum network using advanced fuzz tests. In the wrong hands, this vulnerability (CVE-2020-28362) could have caused a shutdown of the entire Ethereum network. Although the memory-safe Golang module has already undergone extensive security testing, this vulnerability could only be found through fuzz testing.

Ethereum 250x250px

10. What Bugs Can I Find With Fuzzing?

Since 2018, Code intelligence provides a platform for automated fuzz testing. Working closely together with industry and academia, engineers at Code Intelligence were able to find thousands of bugs and extended the reach of modern fuzz testing to a variety of different use cases. Here, you will find an overview of some of the bug classes that the CI team found over the past years. Click here to see the full list.

  • Memory Leaks
    Incorrect management of memory allocation, which is exhausting available memory

  • Injections
    Untrusted input supplied to a program

  • Sensitive Data Exposure
    When an application inadvertently exposes personal data

  • Insecure Deserialization
    Manipulated object injected into the context of a web application

  • Buffer Overflow
    Exceeding the buffer when writing data, which can lead to overwriting adjacent memory locations

  • Use After Free
    No pointing to a valid object that can lead to data corruption, segmentation faults, or general protection faults

  • Data Races
    Accessing the same memory location concurrently in multi-thread process, which can lead to security vulnerabilities

  • Software Crashes
    Failing to function properly and exiting, possible fatal system errors

  • Functional Bugs
    Ceasing to respond to inputs

  • Uncaught Exceptions
    Not properly handling exceptions, which can lead to disrupting program executing

  • Undefined Behavior
    Executing code not prescribed by language specification (C/C++), which can lead to security vulnerabilities

  • And many more

11. What Is the Difference Between Black-Box Fuzzing vs. White-Box Fuzzing?

Much has happened since the early days of fuzzing in the 80s. Among modern solutions, one has to differentiate between white-box and black-box fuzzers. 

What Is Black-Box Fuzzing?  

Black-box fuzzing generates inputs for a target program without knowing its internal behavior or implementation. A black-box fuzzer may generate inputs from scratch, or rely on a static corpus of valid input files to base mutations. Unlike coverage-guided approaches and white-box fuzzing, the corpus does not grow here.

What Is White-Box Fuzzing?  

White-box fuzzers analyze the internal structure of a program, and with each new fuzzing run, they learn to track and maximize code coverage. White-box fuzzers usually use adaptable algorithms and intelligent instrumentation during fuzz testing, which makes them more effective and accurate in detecting vulnerabilities. 

 
The difference between black-box-fuzzing and white-box-fuzzing.

The difference between black-box-fuzzing and white-box-fuzzing.

 

12. What Are Common Fuzz Testing Tools?

Developers can benefit from a whole range of open-source software fuzz testing tools. There are often specialized for specific use cases (e.g. Kernel fuzzing) or programming languages. However, there are some enterprise fuzz tools that are highly effective if you're working in larger development teams or DevOps environments. Usually, they come with more integrations and features, such as automated bug reporting, CI/CD and dev tool integration, REST API fuzzing, or OWASP vulnerability detection.  

Open Source Fuzzers

13. How Can I Get Started With Fuzzing? 

Try to start with an open-source fuzzer like Atheris (for Python) or Jazzer (for Java Testing).  If you’re feeling more comfortable with the testing approach and want to try fuzzing in a more complex environment, there are plenty of enterprise solutions, that come with additional features like reporting, CI/CD/dev tool integration, and API fuzzing. If you want to give such an enterprise solution a go you can try our free demo app with plenty of sample projects.

 

Fuzz Your First Application Today!

Fuzzing your software is now even simpler than unit testing. Set up your own fuzz tests in three commands.

Fuzzing Tutorial

Learn how every developer can secure any Java library in less than 3 minutes. 

Fuzzing Java code using Jazzer.