inbugwetrust/fuzzinator

Paulo Matos

@pmatos

# Saves the execution command as command
[sut.jsc.call.decorate(8)]
property=command
command=echo "${sut.jsc.call:command}"

third: is it the master branch that you are currently using? https://github.com/pmatos/jsc32-fuzz

yes

Renáta Hodován

@renatahodovan

yes, I guess that this is a subprocessPropertyDecorator which save the stdout as byte array

Paulo Matos

@pmatos

This should fix the command issue: pmatos/jsc32-fuzz@e4e3a64

which other issues did you see?

Renáta Hodován

@renatahodovan

yes, this should solve it, since right now your SubprocessJSCCall receives two command arguments, one from the config and one from the database. the first one is a string and the second one is a byte array (I think this way at least)

Paulo Matos

@pmatos

I was actually thinking that you might read the command from the config as a string and then somehow when you import the database, you overwrite the sut['command'] with the bytes and then the validation fails.

I wonder if some property names should be marked as special and not allowed.

This is quite the pitfall.

Renáta Hodován

@renatahodovan

which other issues did you see?

first, your id field is empty: "id" : " ". it should be filled with a unique identifier extracted/built e.g, from the stderr. This id helps fuzzinator to avoid saving the same issue multiple times. probably your regex filter should be double-checked. the problem with this means that fuzzinator cannot distinguish the issues properly and it might misses to save further issues.

Paulo Matos

@pmatos

Yes, I caused this recently...

I removed the RegEx filter and lost the id.

Tuomas (Apple fuzzing guy) suggested I stop fuzzing jsc with asan and when I removed the asan stuff, mistakingly I removed the regex as well.

Need to re-add that.

At the same time, we should implement some sort of warning when the id values are not defined to avoid an empty id.

Renáta Hodován

@renatahodovan

second, the "test" field contains a file path instead of the content of the failing test case. the problem with this, that in a usual fuzzing scenario test cases are only kept until a batch of tests are executed. After this, the old tests are removed and new ones are generated. So, if you save the path of an old test, then you never will be able to reproduce. The solution is to use a FileReaderDecorator like here: https://github.com/renatahodovan/fuzzinator-configs/blob/master/sut/jsc/jsc-grammarinator-cli.ini#L47

Tuomas (Apple fuzzing guy) suggested I stop fuzzing jsc with asan and when I removed the asan stuff, mistakingly I removed the regex as well.

I saw this change, but you removed too much.

https://github.com/pmatos/jsc32-fuzz/commit/d5e4b231023b52fd1b77c187925133c114409eea#diff-2fb0d1e755f0967ab333dc7c78b6a92658ee03ca2627c3affbc081c39ac9e8cbL8

removing the --asan from configs/jsc-build.sh is okay

Paulo Matos

@pmatos

Interesting that the test field contains the path instead of the test. Wonder what I did recently for that to happen... I never used the FileReaderDecorator btw.

Renáta Hodován

@renatahodovan

removing [jsc.build.release] and [jsc.build.debug] doesn't harm either but it has no effect if you don't use them (I might would keep them for later reuse).

Paulo Matos

@pmatos

oh - i know what happened, i removed all the decorators with the word Sanitizer but those were the ones setting the values used by the UniqueIdDecorator.

Renáta Hodován

@renatahodovan

the change in configs/sut-jsc_local.ini should be adapted though:
call.decorate(3)=fuzzinator.call.SanitizerAutomatonFilter should be replaced with call.decorate(3)=fuzzinator.call.RegexAutomatonFilter. The later is the superclass of the previous. RegexAutomatonFilter process the selected stream of the SUT and looks for error patterns. If it founds a matching pattern, than it saves the mathing group name and the matching content as key-value pairs into the issue dictionary. Later, the UniqueIdDecorator will reuse these (and other) fields to generate an identifier (this is the id field which is empty now in your database)
The only difference from SanitizerAutomatonFilter is that SanitizerAutomatonFilter contains predefined patterns for processing sanitized error messages.

fuzzinator.call.SanitizerAnalyzerDecorator won't be neccessary after this but beside executing some extra checks it doesn't change anything if the output is not sanitized

Paulo Matos

@pmatos

Thanks for the input - working on it atm.

I am always worried though that the error patterns in the regex might miss something. Is there a way to avoid that besides writing very thorough patterns?

I should add that armed with this knowledge, I will be on my spare time implementing a fuzzing campaign for racket. I have been holding off to do that because I didn't want to implement a fuzzing framework but now with fuzzinator things got much easier. :)

Renáta Hodován

@renatahodovan

I am always worried though that the error patterns in the regex might miss something. Is there a way to avoid that besides writing very thorough patterns?

well, in short: yes, in long: no :)
you can throw out the whole regex matching thing and save everything that exits with an error code. In this case a random ID will be assigned and EVERYTHING will be saved. Without unification. This could mean hundreds of redundant issues to be validated manually (from my own experience, you don't want to do that)
long: you need to write thorough patterns :)

I should add that armed with this knowledge, I will be on my spare time implementing a fuzzing campaign for racket. I have been holding off to do that because I didn't want to implement a fuzzing framework but now with fuzzinator things got much easier. :)

what is racket? :) is it this language? https://racket-lang.org

Paulo Matos

@pmatos

Yes - that's right.

I am one of the maintainers and fuzzing has been on my list at least since 2019.

John Regehrs team working on fuzzing developed xsmith in racket: https://www.flux.utah.edu/project/xsmith

based on xsmith they developed a fuzzer for racket but not something like fuzzinator, so my first task will be to integrate their racket fuzzer with fuzzinator and see what i get

Renáta Hodován

@renatahodovan

ah thanks for the link! I like the works of Regehr, our research paths crossed a few times in the past. but being honest, I'm a bit lagging behind on reading his papers since my daughter has born :)

Paulo Matos

@pmatos

:) children... they're great, but they do have a tendency to reduce work throughput.

hehe

Renáta Hodován

@renatahodovan

based on xsmith they developed a fuzzer for racket but not something like fuzzinator, so my first task will be to integrate their racket fuzzer with fuzzinator and see what i get

sounds good! tell me if you need help. it's good to see when fuzzinator is used for new targets. you can even upload the configs into fuzzinator-configs when they are ready ;)

Paulo Matos

@pmatos

thanks.

first I need to get JSC working - which i am actually being paid for doing. heheh :)

I have pushed a new revision of my configs. Going to give them a go and see what happens.

Renáta Hodován

@renatahodovan

:) children... they're great, but they do have a tendency to reduce work throughput.

absolutely! this is why I enjoy so much being with the grandparents for a few weeks now, I can go on with my projects :P

they are happy being with their granddaughter and I can do some hobby as well :) a real win-win

first I need to get JSC working - which i am actually being paid for doing. heheh :)

good point :D

Paulo Matos

@pmatos

ah - totally understand you as well. Mine were born in England. Soon after had to move closer to grandparents... which might explain what a portuguese is doing in germany. :)

Renáta Hodován

@renatahodovan

hehe, much closer :D

Paulo Matos

@pmatos

Forgot to mention my wife is German, so we are literally 15mins from grandparents! :)

Renáta Hodován

@renatahodovan

our grandparents live only 100km from us, but because of covid we missed a lot of time last year

Forgot to mention my wife is German, so we are literally 15mins from grandparents! :)

ah, this explains a lot :)

Paulo Matos

@pmatos

@renatahodovan Fuzzinator at the moment does not log anything if an issue is filtered by a decorator. Would you mind if I add a debug log to fuzzinator each time an issue is filtered out?

Where communities thrive

People

Repo info

Activity