How to Secure Complex Microservices in 5 Easy Steps

July 22 2021 | 5 min

Security testing for microservices is particularly hard, as they are highly interconnected. In this article, I will show you how to find a remote code execution vulnerability in an unreleased version of the  German Covid-19 tracing app (CWA), using feedback-based fuzz testing. In 5 steps, I will guide you through the whole process and show you how you can effortlessly build fuzz tests for JVM-based web applications.

In my last blog article, I described why it is so important to improve the reliability and security of web services, now I will show you how to do it.

Bug findingsExample web application: Covid 19 - Contact Tracing App (please note that the expression language injection vulnerabilities were originally found by Alvaro Munoz)


Step 1: Pick your target

In order to provide secure and reliable web services, it is first of all necessary to get familiar with the backend structure, and the architecture of the application. It is important to zoom out and get a rough overview in order to recognize the first interfaces where automated fuzz testing will most likely uncover the most critical vulnerabilities. 

CWA Backend

Example of the backend architecture of the German Covid-19 Tracing App.

I would like to demonstrate this using an example: This scheme illustrates the architecture of the German Covid-19 contact tracing app backend. In Germany, this app is used by large parts of the population to retrace Covid-19 infections and notify people about potential risks. The app also tracks Covid-19 test results and sends them to the user. 

Looking at the architecture overview, we can see that the action of submitting test results is first processed by the submission service, which in turn validates the data via the verification service. This interaction is a perfect target for fuzzing since it processes inputs and is not triggered by cron jobs (i.e. distribution). 


Step 2: Enable your web services for fuzzing

To make full use of modern fuzzing techniques the fuzzer requires feedback from the application under test. In a JVM microservice environment, this is made possible by Java agents. Independent of the binary, they can be added to multiple Java applications, allowing fuzzing of connected services.

To secure a web service, you need to get the backend up and running, so it can be prepared for the fuzz test. With Code Intelligence’s CI Fuzz extension for VS Code, this is pretty easy. First, you need to create a new project. After the project has been initialized, the addition and preparation of the web services can be started. 

add_service-1Adding Services to a fuzz test with CI Fuzz

There are several parameters that you can configure in CI Fuzz. For example, you can choose specified Java packages for the fuzzer, so that only classes from these packages are instrumented. This is quite useful if you don’t want the fuzzer to get lost in external libraries. But to get started, you can simply initialize the project with the default settings.

Setting up the CI Fuzz Java agent doesn't take much time. A big advantage of this fuzzing approach is that it works in any environment where a Java agent can be attached. In the case of the CWA backend, this means that we add the Java agent to the Docker images of the submission and verification service and set the options that we can copy directly from CI Fuzz. Apart from this modification, the existing docker-compose setup remains completely unchanged.

Docekr Compose setupJava agents can be added with very minor modifications


Java Agents

Java agents (often called "Fuzzing Agents") are special Java applications that use the Instrumentation API to modify JVM bytecode during runtime. Thus, they feed information about your application back to your fuzzer. This allows you to measure and maximize code coverage. In most cases, it's enough to add a single option to your Java command. Since many services run in a Docker container, the fuzzing backend is not accessible on localhost. To enable the Java agent to communicate with the fuzzing server,  the Docker host IP must be configured.  Java Agents


Step 3: Configure the fuzz tests

Next, we provide the fuzzer with a description of the API that we want to test. This information is used by the fuzzer to generate relevant HTTP requests that follow the API specification. Such requests have a much higher chance of passing the validation layer of the web service.

For our example, we configure the fuzzer to use the OpenAPI specification of the submission service. This allows the fuzzer to construct valid JSON payloads for the submission data and trigger the interesting parts of the code.

OpenAPI SpecificationTruncated OpenAPI specification

Step 4: Add HTTP requests to get the fuzzer started

To accelerate the fuzzing process it is possible to provide seed requests that the fuzzer uses for further mutation. This step is entirely optional and based on the API definition a set of requests will automatically be generated. Since the fuzzer operates on simple HTTP requests it is also easy to seed the fuzzer with existing tooling such as integration tests, manual API testing tools or even the browser.

For the CWA submission service we use a single seed request with a valid JSON Post body, which was copied from one of the automated tests.

After starting the test we verify that the number of coverage counters increases which is a sure sign that the fuzzer is finding new code paths.


Step 5: Wait until all the bugs have been collected 

Once the fuzzer is done, all the findings and exceptions will be listed with detailed feedback that includes the log, helpful links and the description. Now you can start debugging!

You can find more insights on the CI Fuzz platform by watching our demo video. Click below. 

Watch CI Fuzz Demo

See the Process in Action!

I hope you now have an idea of how fuzz testing can be implemented to secure complex web services. However, I know that reading about the implementation can be a bit tedious. The best thing is to see the process in action. I have recorded a more detailed breakdown of how we secure the Covid-19 tracing app. Click below to see the whole recording.

Get Recording


Recent Posts

One Year of Fuzzing and Fixing Suricata

Autofuzz: Fuzzing Without Writing Fuzz Targets or Harnesses

Fuzzing 101 – The Basics (FAQ)

19 Bugs in Jsoup Found With Jazzer

Share Article

Subscribe to updates