Comparisons of fuzzer

[arcuri82]

Hi all @grebnetiew @ThomasTNO @Stranger6667 @en-milie ,

writing this here to simplify the discussions, and to have it open.

At a certain point, I would like to make a "fair" and "sound" comparison of fuzzers (@Stranger6667 this is somehow related to the discussion we had on Reddit), where all compared fuzzers' authors are directly involved.
This would lead to a possible joint academic publication among us (for whom is interested).

However, what is the status of supporting WFC Authentication? If I understood correctly, WuppieFuzz now supports it. What about Schemathesis and CATS?
This would be needed to easily run experiments on WFD, by using the already existing auth info in WFC format.

This is a very busy period, unfortunately (work and personal reasons). So, don't think anything would happen any time soon before some weeks (there would be several technical details to discuss and agree among us). However, I am writing it now to test out if there is any interest in it.

Let me know.

[grebnetiew]

It would be a nice test case for our WFC auth support to start with :D

We would be particularly interested if this leads to a benchmark framework that is considered 'sound' and/or 'fair' by the broader scientific community. In the past, we have had trouble converging (with article reviewers) on benchmarks that sufficiently represent real world bug-finding performance. In the future, we would like to use such benchmarks as integration tests for new features, fixes etc.