Add fuzz tests

[szadam]

• Code compiles without errors locally
• All tests pass locally
• CI workflows execute properly

• New tests added
• Extended the README/documentation
• All newly added source files have a license
• All newly added source files are referenced in CMake files

This PR is a first of many related to fuzzing - ref. #528

[PatKamin]

Remember to revert this back before merging.

[szadam]

Okey

[bratpiorka]

this should be removed before merge, right?

[szadam]

Yes

[bratpiorka]

please use // clang-format off .... // clang-format on

[lplewa]

Please add --verbose to ctest command (you can remove it just before merge_

I checked your pr localy - and it looks really strange.
First of all i have at the beginning:

This may also happen if the target rejected all inputs we tried so far

Then execution hangs for few mins and then starts running
i tried also just execute ./fuzztest without ctest
then i see something what looks on prompt, during this hang(after a minute or two it starts running)
NEW_FUNC[1/153]:

Not sure if it is expected or not.

Next point: are you sure that we are testing our library? For me we are only testing the test. If we are passing fuzzer flag only to the test, fuzzer only check coverage of this file, while library coverage is not analyzed, to generate better input. am i right?

[szadam]

Please add --verbose to ctest command (you can remove it just before merge_

Done

I checked your pr localy - and it looks really strange. First of all i have at the beginning:

This may also happen if the target rejected all inputs we tried so far

Then execution hangs for few mins and then starts running i tried also just execute ./fuzztest without ctest then i see something what looks on prompt, during this hang(after a minute or two it starts running) NEW_FUNC[1/153]:

Not sure if it is expected or not.

I don't have this rejection message when running the tests locally. This "bug" might be due to different environments or configurations?

Next point: are you sure that we are testing our library? For me we are only testing the test. If we are passing fuzzer flag only to the test, fuzzer only check coverage of this file, while library coverage is not analyzed, to generate better input. am i right?

Yes, we are testing our library. The fuzzing is configured to test API calls defined in umfFuzz.cpp. LibFuzzer needs a clearly defined entry point (main), and in our case, the entry point is LLVMFuzzerTestOneInput, which LibFuzzer calls. The implementation of this function in umfFuzz.cpp determines which parts of the API are tested. LibFuzzer doesn’t function like a coverage tools - it doesn’t "understand" the entire API on its own. We define how to process the fuzzed bytes in the entry point and which API functions are called based on that input.
In each subsequent PR, we will gradually add more API functions to the tests to ensure more comprehensive coverage.

[PatKamin]

Please add --verbose to ctest command (you can remove it just before merge_

I checked your pr localy - and it looks really strange. First of all i have at the beginning:

This may also happen if the target rejected all inputs we tried so far

Then execution hangs for few mins and then starts running i tried also just execute ./fuzztest without ctest then i see something what looks on prompt, during this hang(after a minute or two it starts running) NEW_FUNC[1/153]:

Not sure if it is expected or not.

Next point: are you sure that we are testing our library? For me we are only testing the test. If we are passing fuzzer flag only to the test, fuzzer only check coverage of this file, while library coverage is not analyzed, to generate better input. am i right?

There has to be a test case with the LLVMFuzzerTestOneInput entry point for libFuzzer to work. libFuzzer is not able to check coverage of the whole project. All we can do is create a test that calls as much API functions as possible (but for now let's create a minimal version with just several API functions called).

As to the hangs you get, I don't get any when I run this test locally.

[lplewa]

Please add --verbose to ctest command (you can remove it just before merge_

Done

I checked your pr localy - and it looks really strange. First of all i have at the beginning:

This may also happen if the target rejected all inputs we tried so far

Then execution hangs for few mins and then starts running i tried also just execute ./fuzztest without ctest then i see something what looks on prompt, during this hang(after a minute or two it starts running) NEW_FUNC[1/153]:
Not sure if it is expected or not.

I don't have this rejection message when running the tests locally. This "bug" might be due to different environments or configurations?

Next point: are you sure that we are testing our library? For me we are only testing the test. If we are passing fuzzer flag only to the test, fuzzer only check coverage of this file, while library coverage is not analyzed, to generate better input. am i right?

Yes, we are testing our library. The fuzzing is configured to test API calls defined in umfFuzz.cpp. LibFuzzer needs a clearly defined entry point (main), and in our case, the entry point is LLVMFuzzerTestOneInput, which LibFuzzer calls. The implementation of this function in umfFuzz.cpp determines which parts of the API are tested. LibFuzzer doesn’t function like a coverage tools - it doesn’t "understand" the entire API on its own. We define how to process the fuzzed bytes in the entry point and which API functions are called based on that input. In each subsequent PR, we will gradually add more API functions to the tests to ensure more comprehensive coverage.

from https://llvm.org/docs/LibFuzzer.html

LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.

LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entrypoint (aka “target function”); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM’s [SanitizerCoverage](https://clang.llvm.org/docs/SanitizerCoverage.html) instrumentation.```

https://llvm.org/docs/LibFuzzer.html#using-libfuzzer-as-a-library

[lukaszstolarczuk]

It works for me 👍
I get this may be not the complete solution, but this is just an entry point for us - the issue for fuzz testing is not set as resolved here.

[PatKamin]

Please add --verbose to ctest command (you can remove it just before merge_

Done

I checked your pr localy - and it looks really strange. First of all i have at the beginning:

This may also happen if the target rejected all inputs we tried so far

Then execution hangs for few mins and then starts running i tried also just execute ./fuzztest without ctest then i see something what looks on prompt, during this hang(after a minute or two it starts running) NEW_FUNC[1/153]:
Not sure if it is expected or not.

I don't have this rejection message when running the tests locally. This "bug" might be due to different environments or configurations?

Next point: are you sure that we are testing our library? For me we are only testing the test. If we are passing fuzzer flag only to the test, fuzzer only check coverage of this file, while library coverage is not analyzed, to generate better input. am i right?

Yes, we are testing our library. The fuzzing is configured to test API calls defined in umfFuzz.cpp. LibFuzzer needs a clearly defined entry point (main), and in our case, the entry point is LLVMFuzzerTestOneInput, which LibFuzzer calls. The implementation of this function in umfFuzz.cpp determines which parts of the API are tested. LibFuzzer doesn’t function like a coverage tools - it doesn’t "understand" the entire API on its own. We define how to process the fuzzed bytes in the entry point and which API functions are called based on that input. In each subsequent PR, we will gradually add more API functions to the tests to ensure more comprehensive coverage.

from https://llvm.org/docs/LibFuzzer.html

LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.

LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entrypoint (aka “target function”); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM’s [SanitizerCoverage](https://clang.llvm.org/docs/SanitizerCoverage.html) instrumentation.```

https://llvm.org/docs/LibFuzzer.html#using-libfuzzer-as-a-library

This change should make it: szadam#6

[lukaszstolarczuk]

continued in #572