Golang Fuzzing: Key Improvements in Go Fuzzing (Golang 1.19)

Golang was the first programming language to support fuzzing as a first-class experience in version 1.18. This made it really easy for developers to write fuzz tests. Golang 1.14 introduced native compiler instrumentation for libFuzzer, which enables the use of libFuzzer to fuzz Go code. libFuzzer is one of the most advanced and widely used fuzzing engines and provides the most effective method for Golang Fuzzing.

Below, I want to discuss the various improvements we performed for the libFuzzer mode in Go and show examples of the benefits of Golang fuzzing in version 1.19. In this work, we improved Go’s instrumentation to provide libFuzzer with better signals to guide its mutation and thus explore the tested code more effectively. These improvements are now integrated upstream and will be released in Golang 1.19. 

Golang Fuzzing Support

In Go 1.14, native compiler instrumentation for libFuzzer was added. This code coverage instrumentation within the compiler provides the basis for tools that make use of the feedback. Using -gcflags=all=-d=libfuzzer -buildmode=c-archive as arguments to go build, we can produce a C archive file that contains the instrumented code from the main package and all packages it imports. This archive can then be linked in with libFuzzer manually to produce the final fuzzer. go114-fuzz-build can be used as a wrapper to simplify the process of creating the C-archive. 

In version 1.18 Golang fuzzing was natively included in the standard toolchain, which makes Go the first programming language to make fuzzing a first-class experience. Native Golang fuzzing is built from the ground up and still in an early stage. As a result, fuzing in Golang 1.19 does not match the effectiveness of established fuzzing engines such as libFuzzer.

In the context of this work, we introduced three main improvements to fuzzing support in Golang: supporting libFuzzer’s 8-bit counters, intercepting string compares, and supporting libFuzzer’s value profiling mode. Next, we provide more details and show concrete examples:

Finding More Edges With 8bit Counters

The libFuzzer mode previously used libFuzzer’s extra counters to store the edge coverage counters. Extra counters are a feature that enables the instrumentation to store a counters array in a dedicated section called __libfuzzer_extra_counters. libFuzzer then interprets these counters in the same way it does for 8-bit counters to compute the feature's metrics that are used to tell whether the currently executed input is interesting. The exact semantics of extra counters is up to the instrumentation, and they are therefore only reported as features. As a result, it is not possible to see whether libFuzzer actually reached a new edge. We changed this to use libFuzzer’s 8-bit counters instead.

This improves the coverage instrumentation in libFuzzer mode in three ways:

• 8-bit counters are supported on all platforms, including macOS and Windows, with all relevant versions of libFuzzer, whereas extra counters are a Linux-only feature that only recently received support on Windows
• Newly covered edges are now properly reported as new coverage by libFuzzer, not only as new features. This corresponds to the cov metric reported by libFuzzer.
• The NeverZero strategy developed by the AFL++ team is used to ensure that coverage counters never become 0 again after having been positive once. This resolves issues encountered when fuzzing loops with iteration counts that are multiples of 256 (e.g., larger powers of two). Without this policy, an edge appears to be not executed in this special case, and we have a potential to lose a bit of information. The experiments performed by the AFL++ team showed that the NeverZero strategy is very effective and improves AFL++ in terms of coverage and speed (the seed selection now takes into account edges that were hidden before). 
  
  

Intercepting String Comparisons

Often, the code under test contains string comparisons of various forms. Obviously, fuzzers are very unlikely to generate test cases that pass these checks by solely relying on code coverage. To efficiently handle these cases, libFuzzer has callbacks that can be used by the instrumentation to report string comparisons encountered while executing the code. The compared strings will then be added to libFuzzer’s table of recent compares, which feeds future mutations performed by the fuzzer and thus allow it to reach into branches guarded by string comparisons. We augmented the libFuzzer build mode so that direct string compares, as well as calls to string comparison functions such as strings.EqualFold, are intercepted and the corresponding libFuzzer callbacks are invoked with the corresponding arguments. The list of methods to intercept is maintained in cmd/compile/internal/walk/expr.go and can easily be extended to cover more standard library functions in the future. 

Let’s have a look at an example:

func FuzzStringCompare(data []byte) int {
  if len(data) < 9 {
    return 0
  }
  str1 := string(data[:8])
  str2 := string(data[8:])
  if str1 == "Awesome" && strings.Compare(str2, "Fuzzing!") == 0 {
    panic("We found a bug!")
  }
  return 0
}

go114-fuzz-build -o target.a -func FuzzStringCompare .
clang -fsanitize=fuzzer target.a -o fuzzer
./fuzzer

INFO: Seed: 1843763752
INFO: Loaded 1 modules   (250 inline 8-bit counters): 250 [0xecad68, 0xecae62), 
INFO: Loaded 1 PC tables (250 PCs): 250 [0xc00006a000,0xc00006afa0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2        INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 25Mb
#717      NEW    cov: 4 ft: 4 corp: 2/12b lim: 11 exec/s: 0 rss: 25Mb L: 11/11 MS: 5 CrossOver-InsertByte-ChangeByte-ChangeBit-CMP- DE: "@\x00\x00\x00\x00\x00\x00\x00"-
#721      REDUCE cov: 4 ft: 4 corp: 2/11b lim: 11 exec/s: 0 rss: 25Mb L: 10/10 MS: 4 ShuffleBytes-ChangeByte-ChangeByte-EraseBytes-
#794      REDUCE cov: 4 ft: 4 corp: 2/10b lim: 11 exec/s: 0 rss: 25Mb L: 9/9 MS: 3 CopyPart-ShuffleBytes-EraseBytes-
#1245     REDUCE cov: 7 ft: 7 corp: 3/19b lim: 14 exec/s: 0 rss: 25Mb L: 9/9 MS: 1 CMP- DE: "Awesome "-
[...]
#1048576  pulse  cov: 9 ft: 9 corp: 5/121b lim: 4096 exec/s: 262144 rss: 32Mb
panic: We found a bug!
goroutine 17 [running, locked to thread]:
golang.org/dl/gotip/code/kyakdan/go-exmples.FuzzStringCompare(...)
        golang.org/dl/gotip/code/kyakdan/go-exmples/main.go:48
main.LLVMFuzzerTestOneInput(...)
       ./main.652335763.go:21
==257079== ERROR: libFuzzer: deadly signal
    #0 0x4aefb0 in __sanitizer_print_stack_trace (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x4aefb0)
    #1 0x45b2b8 in fuzzer::PrintStackTrace() (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x45b2b8)
    #2 0x440403 in fuzzer::Fuzzer::CrashCallback() (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x440403)
    #3 0x7fe46d17c3bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0x143bf)
    #4 0x506fe0 in runtime.raise.abi0 runtime/sys_linux_amd64.s:158

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 4 ChangeByte-EraseBytes-PersAutoDict-CMP- DE: "Awesome "-"Fuzzing!"-; base unit: 84b4aeab1b0456c420d4600f53b944cf29246d21
0x41,0x77,0x65,0x73,0x6f,0x6d,0x65,0x20,0x46,0x75,0x7a,0x7a,0x69,0x6e,0x67,0x21,
Awesome Fuzzing!
artifact_prefix='./'; Test unit written to ./crash-dd107abbf60f67c533ff7aecb116ce483fc4facf
Base64: QXdlc29tZSBGdXp6aW5nIQ==

Supporting libFuzzer Value Profiling Mode for Integer Compares 

libFuzzer provides a special mode known as “value profiling” in which it tracks the bit-wise progress made by the fuzzer in satisfying tracked comparisons. libFuzzer has special hooks to intercept various types of integer comparisons. In this mode, libFuzzer uses the value of the return address in its hooks to distinguish the progress for different comparisons.

The original implementation of the interception for integer comparisons in Go simply called the libFuzzer hooks from a function written in Go assembly. As a result, the libFuzzer hooks thus always saw the same return address (i.e., the address of the call instruction in the assembly snippet) and thus could not distinguish individual comparisons anymore. This meant that all integer comparisons looked to libFuzzer as a single comparison, and thus drastically reduced the usefulness of value profiling.

We fixed this issue by using an assembly trampoline that injects synthetic but valid return addresses (we call them fake PCs) before calling the libFuzzer hook, otherwise preserving the calling convention of the respective platform. For X86_64 the return address is pushed on the stack and for ARM64 we use the link register. These fake PCs are generated deterministically based on the location of the compare instruction in the IR representation. The assembly trampoline idea was pioneered by our open-source Java Fuzzer Jazzer.

Let’s have a look at an example that shows the improved value profiling in action:

func encrypt(n uint64) uint64 {
  return n ^ 0x1122334455667788
}

func FuzzXorEncrypt(data []byte) int {
  if len(data) < 16 {
    return 0
  }
  n := binary.BigEndian.Uint64(data[:8])
  n2 := binary.BigEndian.Uint64(data[8:16])

  if encrypt(n) == 5788627691251634856 && encrypt(n2) == 6293579535917519017 {
    panic("XOR with a constant is not a secure encryption ;-)")
  }
  return 0
}

The example uses the encrypt function to transform the input and then compares the results with two constants. The panic can only be triggered when both checks are fulfilled. To build this code we do the following:

go114-fuzz-build -o target.a -func FuzzXorEncrypt .
clang -fsanitize=fuzzer target.a -o fuzzer
./fuzzer -use_value_profile=1

INFO: Seed: 2476973394
INFO: Loaded 1 modules   (246 inline 8-bit counters): 246 [0xecad68, 0xecae5e), 
INFO: Loaded 1 PC tables (246 PCs): 246 [0xc00006c000,0xc00006cf60), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2       INITED cov: 2 ft: 8 corp: 1/1b exec/s: 0 rss: 25Mb
#47      NEW    cov: 2 ft: 9 corp: 2/4b lim: 4 exec/s: 0 rss: 25Mb L: 3/3 MS: 5 CrossOver-CopyPart-EraseBytes-CopyPart-CrossOver-
#511     NEW    cov: 2 ft: 10 corp: 3/11b lim: 8 exec/s: 0 rss: 25Mb L: 7/7 MS: 4 ChangeBinInt-ChangeBit-InsertRepeatedBytes-CopyPart-
#821     NEW    cov: 2 ft: 11 corp: 4/22b lim: 11 exec/s: 0 rss: 25Mb L: 11/11 MS: 5 CrossOver-ChangeBinInt-InsertByte-ChangeByte-CrossOver-
#825     REDUCE cov: 2 ft: 11 corp: 4/20b lim: 11 exec/s: 0 rss: 25Mb L: 9/9 MS: 4 CopyPart-ChangeBinInt-CMP-EraseBytes- DE: "\xff\xff\xff\xff\xff\xff\xff "-
[...]
#155400  REDUCE cov: 4 ft: 141 corp: 119/2695b lim: 958 exec/s: 0 rss: 25Mb L: 16/495 MS: 4 ShuffleBytes-ChangeBinInt-ShuffleBytes-EraseBytes-
#157796  NEW    cov: 4 ft: 142 corp: 120/2712b lim: 976 exec/s: 0 rss: 25Mb L: 17/495 MS: 1 ChangeBit-
#161697  REDUCE cov: 4 ft: 142 corp: 120/2711b lim: 1012 exec/s: 161697 rss: 25Mb [...]
#1283495 REDUCE cov: 5 ft: 272 corp: 241/11724b lim: 4096 exec/s: 183356 rss: 26Mb L: 16/4079 MS: 3 EraseBytes-ChangeBit-InsertByte-
#1396131 REDUCE cov: 5 ft: 272 corp: 241/11722b lim: 4096 exec/s: 174516 rss: 26Mb L: 16/4079 MS: 1 EraseBytes-
#1396137 REDUCE cov: 5 ft: 272 corp: 241/11721b lim: 4096 exec/s: 174517 rss: 26Mb L: 16/4079 MS: 1 EraseBytes-
panic: XOR with a constant is not a secure encryption ;-)

goroutine 17 [running, locked to thread]:
golang.org/dl/gotip/code/kyakdan/go-exmples.FuzzXorEncrypt(...)
      golang.org/dl/gotip/code/kyakdan/go-exmples/main.go:114
main.LLVMFuzzerTestOneInput(...)
      ./main.3429615431.go:21
==258863== ERROR: libFuzzer: deadly signal
    #0 0x4aefb0 in __sanitizer_print_stack_trace (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x4aefb0)
    #1 0x45b2b8 in fuzzer::PrintStackTrace() (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x45b2b8)
    #2 0x440403 in fuzzer::Fuzzer::CrashCallback() (/home/khaled/code/kyakdan/go-exmples/fuzzer+0x440403)
    #3 0x7efc89ef23bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0x143bf)
    #4 0x506fe0 in runtime.raise.abi0 runtime/sys_linux_amd64.s:158

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 1 ChangeBit-; base unit: 5721add88aec96d9ec47a5a56a1d16c90b2ad332
0x41,0x77,0x65,0x73,0x6f,0x6d,0x65,0x20,0x46,0x75,0x7a,0x7a,0x69,0x6e,0x67,0x21,
Awesome Fuzzing!
artifact_prefix='./'; Test unit written to 
./crash-dd107abbf60f67c533ff7aecb116ce483fc4facf
Base64: QXdlc29tZSBGdXp6aW5nIQ==

Closing Thoughts

In this blog post, I have given an overview of our recent work on improving Golang fuzzing, which will be released in Golang 1.19. We have performed a quick run for all Go projects in OSS-Fuzz and found 7 new bugs in them with the improved instrumentation. So far, some Go projects in OSS-Fuzz already use our fork containing the patches. All projects will then automatically benefit from the improvements when Golang 1.19 has been released. Next steps for Go include developing new bug detectors to find more interesting bugs. 

If you want to have a look at the improvements we made in more detail, you can check all our merged pull requests. If you encounter issues with the new instrumentation or have ideas for bug detectors or better instrumentation, we’d love to chat about that.