testing: add fuzz test support

[katiehockman]

This proposal is to add fuzz test support to Go. This will add a new testing.F type, support FuzzFoo functions in _test.go files, and add new go command behavior.

A design draft has already been published and iterated on based on feedback from the Go community. This is the next step to propose that this design draft become a language feature.

This feature will be considered experimental in Go 1.18, and the API will not be covered by the Go 1 compatibility promise yet. The functionality that goes into this release is expected to have bugs and be missing features, but should serve as a proof-of-concept for which Go developers can experiment and provide feedback. Since this will be an experimental feature to start, we also expect there will be room for growth for the mutator and fuzzing engine in future Go releases.

Below are the parts of the design draft which will not make it into 1.18, and will be considered later on:

• support for fuzzing with -race and -msan
• support for fuzzing with -keepfuzzing
• deduplication of similar crashes caused by different mutations, which would be a prerequisite to implementing -keepfuzzing (to reduce noise)
• allowing special options while fuzzing (e.g. maximum input size)
• dictionary support
• customizable coverage instrumentation while fuzzing (e.g. to only instrument certain packages or files)
• custom generators for the mutator
• structured fuzzing support for struct and non-primitive types
• [Stretch goal for 1.18] structured fuzzing support for primitive types other than []byte (e.g. string, int, float64)

Edit: This previously said 1.17, but it was not merged to master to make it into 1.17, so this has been updated to say "1.18" instead.

[robpike]

How can we keep the fuzzing corpora small, and the fuzzing process efficient and on point? Although fuzzing is good at finding certain classes of bugs, it is very expensive in CPU and storage and cost/benefit ratio remains unclear. I worry about wasting energy and filling up git repos with testdata noise. Is there a way to make the testing focused and result-driven? Also, will we wake up one day and find fuzzing is a mandatory part of all builds? That would be terrifying.

Apologies if I'm late to the party; I did not comment on the original proposal.

[FiloSottile]

How can we keep the fuzzing corpora small, and the fuzzing process efficient and on point? Although fuzzing is good at finding certain classes of bugs, it is very expensive in CPU and storage and cost/benefit ratio remains unclear.

The track record of go-fuzz provides pretty amazing evidence that fuzzing is good at finding bugs that humans had not found. In my experience, just a few CPU minutes of fuzzing can be extremely effective at the margin. There aren't a lot of other ways to turn fixed amounts of CPUs (as opposed to dynamic, runtime checks) into bugs caught.

I worry about wasting energy and filling up git repos with testdata noise. Is there a way to make the testing focused and result-driven?

Yeah, filling repositories with random-looking stuff is definitely undesirable. To be clear, in this design the mutator-generated corpus is not in testdata, we agree that it would be way too noisy to check in.

Samples are only placed in testdata when they resulted in a test failure, and then it's up to the developer to check them in as a regression test, or turn them into a more full-fledged unit test.

Also, will we wake up one day and find fuzzing is a mandatory part of all builds? That would be terrifying.

I'm not sure what you mean with being a part of all builds. If you mean running as part of go build, definitely not.

The expensive, open-ended part of fuzzing won't even be a default part of go test: without the -fuzz flag only seed and testdata corpora will be tested, and the mutator won't run. (That would make tests non-deterministic, aside from resources concerns.)

[robpike]

Thanks, Filo. Your point about how we store stuff is important, and I missed that.

I understand the value of fuzzing, and my grumpy old man concerns are not an attempt to reduce that. (I will say that some of the bugs it finds are not worth fixing, and certainly not the cost of finding, although it's very hard to make that case.) But I have three general concerns:

1. A concentration of testing on fuzzing. It's great, but it's not the only way to find bugs, although some act as if it is.

2. The massive computational resources it consumes. It's directed, but not that far off a million monkeys at a million keyboards.

3. Its adoption as part of the process. This is where the proposal to incorporate it needs attention. The technical process and the culture must make it clear that (1) and (2) mean that it cannot be part of the standard build process.

Go build will surely not run it always, but organizations will. Being automated, companies will require it, just as they require zero lint errors even as we protest that lint is not perfect.

I am just expressing caution, care, and an eye on the big picture here. I am not saying we shouldn't do this, just that we should do it with all considerations in mind.

[everestmz]

This is awesome - really excited to see native Go fuzzing in the proposal stage now.

Hopefully it's not too early to make a suggestion for the actual implementation of this, but it would be super useful if the testing.F type was an interface, rather than a struct like testing.T/B are.

I think fuzzing is in sort of a unique spot among the supported testing types in that there's no one correct way to fuzz test your code. Having the ability to build a type that implements the testing.F interface would make it simple to swap out fuzzing backends, enabling the community to take advantage of all the awesome open source work out there by building integrations to things like AFL++.

There's already been interest in something like this from the original Reddit thread mentioned on the proposal document (like this comment here), and I think it would be a missed opportunity to not allow developers to hook these functions into property-based testing tools (like Gopter) to get some minimal testing done at the go test stage, while allowing the same test to later be fuzzed to discover deeper bugs.

There are probably details I've missed here and I'd like to hear some other opinions on this, but given all the potential use cases I think it's worth some discussion.

[FiloSottile]

1. A concentration of testing on fuzzing. It's great, but it's not the only way to find bugs, although some act as if it is.

Agreed, I like to say that fuzzing finds bugs at the margin. It's why we are interested in it as the Security team: bugs caught at the margin are ones that don't make it into production to become vulnerabilities. Just like it finds bugs that are often not found in other ways, it can't find bugs that other ways can.

We'll do our best to communicate to developers in the coming months about what fuzzing can (and can't) do for them, and we'd like to hear it if that content doesn't push in the right direction.

2. The massive computational resources it consumes. It's directed, but not that far off a million monkeys at a million keyboards.
3. Its adoption as part of the process. This is where the proposal to incorporate it needs attention. The technical process and the culture must make it clear that (1) and (2) mean that it cannot be part of the standard build process.

Indeed, this is the hardest part: creating a culture in the ecosystem that makes the practice successful. We want fuzzing to be part of the development—not build or security—process: make a change to the relevant code, run go test -fuzz with the locally cached corpus that can already hit most code branches, find a bug quickly, debug-fix-verify with go test, turn the crasher into a unit test or check it in as a regression test.

There is probably also space for fuzzing in CI, like OSS-Fuzz demonstrated and so that the corpus can be centralized, but since it's non-deterministic it can't meaningfully block a build. Hopefully the CLI choices, like not having a fixed fuzzing time, nudge in that direction.

Likewise, I think writing will be the best tool we have to shape usage, and welcome feedback on future drafts!

[bcmills]

@everestmz

it would be super useful if the testing.F type was an interface, rather than a struct like testing.T/B are.

Why would such an interface type need to be defined in the testing package itself? Keeping testing.F a concrete type allows for future methods to be added and documented, and packages that want to abstract over different F-like implementations can already defined their own interface for the specific methods they need.

[JAicewizard]

turn the crasher into a unit test or check it in as a regression test.

Absolute wild wild dream, but... wouldn't it be amazing if there would be an option to transform bugs found with fuzzing into unit test cases automatically? It could be as simple as copying the function, give is a unique name, and replace testData := {source of random data} with testData := {the data that makes the test fail}.

This would make it very clear that fuzzing is generating data that make your test fail, and in itself not a way to test your code. Its a random edg-case generator, not a validator.

[bcmills]

The flow of control in the design doc seems a bit off to me. The Fuzz method in particular carries a lot of type-state invariants:

• “Only one call to Fuzz is allowed per fuzz target, and any subsequent calls will panic.”
• and, Add “cannot be invoked after or within the Fuzz function.”

Making Fuzz a method seems to invite user code to try to perform setup before the call to Fuzz and/or cleanup afterward. (Is that the intent?)

But then I wonder if there is a cleaner alternative, especially given the (*T).Cleanup methods added in #32111. The restriction to a single Fuzz invocation per test makes me think that it functions more like a constructor-function than a method call on an existing object. Would it make sense to make it a constructor method on a *T, returning an explicit object on which Add could then be called? That would make it clearer that the fuzz function should not call Add, since the user would have to go out of their way to even have the object with the Add method in scope at that point.

I'm imagining something like:

package testing

func (*T) NewFuzzer(ff interface{}) *F
func (*F) Add(args interface{})

That would make the example in the doc more like:

func TestFuzzMarshalFoo(t *testing.T) {
	f := t.NewFuzzer(func(t *testing.T, a string, num *big.Int) {
		t.Parallel() // seed corpus tests can run in parallel
		if num.Sign() <= 0 {
			t.Skip() // only test positive numbers
		}
		val, err := MarshalFoo(a, num)
		if err != nil {
			t.Skip()
		}
		if val == nil {
			t.Fatal("MarshalFoo: val == nil, err == nil")
		}
		a2, num2, err := UnmarshalFoo(val)
		if err != nil {
			t.Fatalf("failed to unmarshal valid Foo: %v", err)
		}
		if a2 == nil || num2 == nil {
			t.Error("UnmarshalFoo: a==nil, num==nil, err==nil")
		}
		if a2 != a || !num2.Equal(num) {
			t.Error("UnmarshalFoo does not match the provided input")
		}
	})

	// Seed the initial corpus
	f.Add("cat", big.NewInt(1341))
	f.Add("!mouse", big.NewInt(0))
}

If the -fuzz flag is set, the additional fuzzing would occur after TestFuzzMarshalFoo returns but before its Cleanup callbacks are executed.

[katiehockman]

@JAicewizard

wouldn't it be amazing if there would be an option to transform bugs found with fuzzing into unit test cases automatically?

There were discussions about code generation in some of the first designs, but it was eventually decided that it was a complexity without much gain, especially since we can gain the same benefits in a simpler way. With this design, new crashes are put into testdata and will automatically run as a regression test on the next go test run (even when -fuzz isn't used).

And if someone would prefer to put this crash into the code instead of using the one in testdata, they have a couple options:

1. They can use f.Add with the existing fuzz target (which works basically the same way as it would if it were in testdata, its just in the code instead)
2. They can copy-paste the Fuzz target and make slight alterations to the code to make it a unit test. We intentionally designed fuzz targets to look very similar to unit tests, which allows fuzz targets to be easily translated into unit tests, but also allows unit tests to be easily translated into fuzz targets.

[josharian]

Fuzz functions will change over time, as all code does. If/when they change, the crashes in testdata may no longer be effectively testing against regressions. It seems to me that crashes need to be reified in regular tests.

That said, a trained corpus (not just crashes) can make fuzzing much more effective at finding new issues. I understand not wanting to pollute a repo with a corpus, but I think we should design an intended way for people to keep and share corpora. I have used git submodules in the past reasonably hapiily, but that's git-specific (and submodules are generally frustrating to work with). I'm not sure what the answer is here, but I think at least trying for an answer is an important part of the fuzzing big picture.

[dbrumley]

A concentration of testing on fuzzing. It's great, but it's not the only way to find bugs, although some act as if it is.

I think part of the attraction is not just it finds bugs, but also how it integrates with developer workflow. The nice thing about fuzzing is that each bug has a witness input. That input, and the ability to debug with your own tools, makes fuzzing-as-a-process quite nice.

That being said, definitely no silver bullets here.

The massive computational resources it consumes. It's directed, but not that far off a million monkeys at a million keyboards.

This also relates to the proposal "This type of testing works best when able to run more mutations quickly, rather than fewer mutations intelligently.".

AFL/libfuzzer really focus on iteration speed, but there are other approaches like symbolic execution which are slow, but generate new coverage (more or less) by definition on each iteration. Less fast million monkeys; more like applied model checking using heavy-weight SMT/SAT solvers. Most research, and some products, pair the two together. I know for a fact Mayhem and Mechanical Phish did this in the cyber grand challenge, for instance.

The great thing about the proposal is the interface and build process is standardized. I view this proposal as relevant beyond fuzzing to the broader set of dynamic analysis tools.