call.decorate(3)=fuzzinator.call.SanitizerAutomatonFilter should be replaced with call.decorate(3)=fuzzinator.call.RegexAutomatonFilter. The later is the superclass of the previous. RegexAutomatonFilter process the selected stream of the SUT and looks for error patterns. If it founds a matching pattern, than it saves the mathing group name and the matching content as key-value pairs into the issue dictionary. Later, the UniqueIdDecorator will reuse these (and other) fields to generate an identifier (this is the id field which is empty now in your database)The only difference from SanitizerAutomatonFilter is that SanitizerAutomatonFilter contains predefined patterns for processing sanitized error messages.
fuzzinator.call.SanitizerAnalyzerDecorator won't be neccessary after this but beside executing some extra checks it doesn't change anything if the output is not sanitized
I am always worried though that the error patterns in the regex might miss something. Is there a way to avoid that besides writing very thorough patterns?
well, in short: yes, in long: no :)
you can throw out the whole regex matching thing and save everything that exits with an error code. In this case a random ID will be assigned and EVERYTHING will be saved. Without unification. This could mean hundreds of redundant issues to be validated manually (from my own experience, you don't want to do that)
long: you need to write thorough patterns :)
I should add that armed with this knowledge, I will be on my spare time implementing a fuzzing campaign for racket. I have been holding off to do that because I didn't want to implement a fuzzing framework but now with fuzzinator things got much easier. :)
what is racket? :) is it this language? https://racket-lang.org
based on xsmith they developed a fuzzer for racket but not something like fuzzinator, so my first task will be to integrate their racket fuzzer with fuzzinator and see what i get
sounds good! tell me if you need help. it's good to see when fuzzinator is used for new targets. you can even upload the configs into fuzzinator-configs when they are ready ;)
Forgot to mention my wife is German, so we are literally 15mins from grandparents! :)
ah, this explains a lot :)
do you mean to add debug log to the regexautomatonfilter or to every filter? since if you mean the latter, then it'd mean a print after every SUT call: a print if an issue is kept and another if it's thrown away. plus, every print could mean a bigger blob of text (stdout + stderr) which is quite hard to process.
but I know your motivation (ensuring catching all the issues). what I used to apply to refine my regexes in case of a new sut is adding (prepending) the very simple versions of the known patterns to the regex list. E.g., if a SUT applies a complicate ASSERT pattern, then you define a regex with the ASSERT keyword only, then a more complex one with all the details. This construct ensures catching every output containing the ASSERT keyword and refines it if your complex pattern is correct. If you receive an error with the ASSERT keyword only, then you know that your complex pattern should be improved. However, this must be a debug-only solution until your regexes are finalized, since if you save an issue with the ASSERT keyword as id then in the next similar use-case the new issue won't be saved since it's id can be the ASSERT keyword again.
NonIssue instance:https://github.com/renatahodovan/fuzzinator/blob/master/fuzzinator/job/fuzz_job.py#L55
(
NonIssue is basically an issue that was filtered out, but it contains all of its original fields)
-vv or something similar so I don't think it would be a problem to have upstream.
-v for debug logs, but it's used to feedback about "bigger" events, like new jobs, finished jobs, SUT timeouts or displaying the error messages of the caught issues. Printing details about "almost" issues should be reported on a lower log level, maybe on trace (although if this level doesn't exist in Fuzzinator yet) Furthermore, if we log about regexautomatonfilter and exit code filter, then we should support others, too. I still believe that the best way of doing this would be to print the content of a NonIssue in fuzz_job, after the SUT returned the result of the test case.
{{foo}} come from?
fuzzer_1 | Exception in <fuzzinator.job.validate_job.ValidateJob object at 0xf5183350>: decorator() missing 1 required positional argument: 'filename'
fuzzer_1 | Traceback (most recent call last):
fuzzer_1 | File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/controller.py", line 413, in _run_job
fuzzer_1 | for issue in job.run():
fuzzer_1 | File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/job/validate_job.py", line 29, in run
fuzzer_1 | _, new_issues = self.validate()
fuzzer_1 | File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/job/validate_job.py", line 33, in validate
fuzzer_1 | sut_call, sut_call_kwargs = config_get_callable(self.config, 'sut.' + self.sut_name, ['validate_call', 'reduce_call', 'call'])
fuzzer_1 | File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/config.py", line 74, in config_get_callable
fuzzer_1 | entity = decorator(entity)
fuzzer_1 | File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/call/callable_decorator.py", line 32, in __call__
fuzzer_1 | return self.decorator(*self.decorator_args, **self.decorator_kwargs)(callable)
fuzzer_1 | TypeError: decorator() missing 1 required positional argument: 'filename'This looks like I am missing filename for validation. But I have no idea why or where it's needed. Is there any way to find more information?
filename is probably missing from the file_writer_decorator:
https://github.com/renatahodovan/fuzzinator/blob/master/fuzzinator/call/file_writer_decorator.py#L45