which other issues did you see?
id field is empty:
"id" : " ". it should be filled with a unique identifier extracted/built e.g, from the stderr. This id helps fuzzinator to avoid saving the same issue multiple times. probably your regex filter should be double-checked. the problem with this means that fuzzinator cannot distinguish the issues properly and it might misses to save further issues.
"test"field contains a file path instead of the content of the failing test case. the problem with this, that in a usual fuzzing scenario test cases are only kept until a batch of tests are executed. After this, the old tests are removed and new ones are generated. So, if you save the path of an old test, then you never will be able to reproduce. The solution is to use a
FileReaderDecoratorlike here: https://github.com/renatahodovan/fuzzinator-configs/blob/master/sut/jsc/jsc-grammarinator-cli.ini#L47
Tuomas (Apple fuzzing guy) suggested I stop fuzzing jsc with asan and when I removed the asan stuff, mistakingly I removed the regex as well.
I saw this change, but you removed too much.
call.decorate(3)=fuzzinator.call.SanitizerAutomatonFiltershould be replaced with
call.decorate(3)=fuzzinator.call.RegexAutomatonFilter. The later is the superclass of the previous.
RegexAutomatonFilterprocess the selected stream of the SUT and looks for error patterns. If it founds a matching pattern, than it saves the mathing group name and the matching content as key-value pairs into the issue dictionary. Later, the UniqueIdDecorator will reuse these (and other) fields to generate an identifier (this is the
idfield which is empty now in your database)
fuzzinator.call.SanitizerAnalyzerDecoratorwon't be neccessary after this but beside executing some extra checks it doesn't change anything if the output is not sanitized
I am always worried though that the error patterns in the regex might miss something. Is there a way to avoid that besides writing very thorough patterns?
well, in short: yes, in long: no :)
you can throw out the whole regex matching thing and save everything that exits with an error code. In this case a random ID will be assigned and EVERYTHING will be saved. Without unification. This could mean hundreds of redundant issues to be validated manually (from my own experience, you don't want to do that)
long: you need to write thorough patterns :)
I should add that armed with this knowledge, I will be on my spare time implementing a fuzzing campaign for racket. I have been holding off to do that because I didn't want to implement a fuzzing framework but now with fuzzinator things got much easier. :)
what is racket? :) is it this language? https://racket-lang.org
based on xsmith they developed a fuzzer for racket but not something like fuzzinator, so my first task will be to integrate their racket fuzzer with fuzzinator and see what i get
sounds good! tell me if you need help. it's good to see when fuzzinator is used for new targets. you can even upload the configs into fuzzinator-configs when they are ready ;)
first I need to get JSC working - which i am actually being paid for doing. heheh :)
good point :D