Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Paulo Matos
    @pmatos
    :) children... they're great, but they do have a tendency to reduce work throughput.
    hehe
    Renáta Hodován
    @renatahodovan

    based on xsmith they developed a fuzzer for racket but not something like fuzzinator, so my first task will be to integrate their racket fuzzer with fuzzinator and see what i get

    sounds good! tell me if you need help. it's good to see when fuzzinator is used for new targets. you can even upload the configs into fuzzinator-configs when they are ready ;)

    Paulo Matos
    @pmatos
    thanks.
    first I need to get JSC working - which i am actually being paid for doing. heheh :)
    I have pushed a new revision of my configs. Going to give them a go and see what happens.
    Renáta Hodován
    @renatahodovan

    :) children... they're great, but they do have a tendency to reduce work throughput.

    absolutely! this is why I enjoy so much being with the grandparents for a few weeks now, I can go on with my projects :P

    they are happy being with their granddaughter and I can do some hobby as well :) a real win-win

    first I need to get JSC working - which i am actually being paid for doing. heheh :)

    good point :D

    Paulo Matos
    @pmatos
    ah - totally understand you as well. Mine were born in England. Soon after had to move closer to grandparents... which might explain what a portuguese is doing in germany. :)
    Renáta Hodován
    @renatahodovan
    hehe, much closer :D
    Paulo Matos
    @pmatos
    Forgot to mention my wife is German, so we are literally 15mins from grandparents! :)
    Renáta Hodován
    @renatahodovan
    our grandparents live only 100km from us, but because of covid we missed a lot of time last year

    Forgot to mention my wife is German, so we are literally 15mins from grandparents! :)

    ah, this explains a lot :)

    Paulo Matos
    @pmatos
    @renatahodovan Fuzzinator at the moment does not log anything if an issue is filtered by a decorator. Would you mind if I add a debug log to fuzzinator each time an issue is filtered out?
    Renáta Hodován
    @renatahodovan

    do you mean to add debug log to the regexautomatonfilter or to every filter? since if you mean the latter, then it'd mean a print after every SUT call: a print if an issue is kept and another if it's thrown away. plus, every print could mean a bigger blob of text (stdout + stderr) which is quite hard to process.

    but I know your motivation (ensuring catching all the issues). what I used to apply to refine my regexes in case of a new sut is adding (prepending) the very simple versions of the known patterns to the regex list. E.g., if a SUT applies a complicate ASSERT pattern, then you define a regex with the ASSERT keyword only, then a more complex one with all the details. This construct ensures catching every output containing the ASSERT keyword and refines it if your complex pattern is correct. If you receive an error with the ASSERT keyword only, then you know that your complex pattern should be improved. However, this must be a debug-only solution until your regexes are finalized, since if you save an issue with the ASSERT keyword as id then in the next similar use-case the new issue won't be saved since it's id can be the ASSERT keyword again.

    however, if you really would like to print every thrown away issues, then you need to put a print here in case of the issue is a NonIssue instance:
    https://github.com/renatahodovan/fuzzinator/blob/master/fuzzinator/job/fuzz_job.py#L55
    (NonIssue is basically an issue that was filtered out, but it contains all of its original fields)
    Paulo Matos
    @pmatos
    I was thinking of a logger.debug(...) in RegexAutomatonFilter and ExitCodeFilter only if an issue is filtered out.
    This would only print if we enabled -vv or something similar so I don't think it would be a problem to have upstream.
    What do you think?
    Renáta Hodován
    @renatahodovan
    hm, we already have -v for debug logs, but it's used to feedback about "bigger" events, like new jobs, finished jobs, SUT timeouts or displaying the error messages of the caught issues. Printing details about "almost" issues should be reported on a lower log level, maybe on trace (although if this level doesn't exist in Fuzzinator yet) Furthermore, if we log about regexautomatonfilter and exit code filter, then we should support others, too. I still believe that the best way of doing this would be to print the content of a NonIssue in fuzz_job, after the SUT returned the result of the test case.
    Paulo Matos
    @pmatos
    OK - happy to add that to the NonIssue in fuzz_job and see how far I go.
    I don't think there's a log level called trace: https://docs.python.org/3/howto/logging.html
    DEBUG is the lowest one.
    Renáta Hodován
    @renatahodovan
    I know, but we can define custom levels. We have a not yet published implementation to support the TRACE level (and other common features) in all the -inators project. Since now it's needed I'll rehash it and will notify you if it can be used. Until then, you can use the DEBUG level locally to debug your setup.
    Paulo Matos
    @pmatos
    Thanks.
    By the way, in the bug reports, the ones where we use jinja, where do the variables {{foo}} come from?
    Can I access anything for which I defined in a property decorator?
    Didn't get validation to work properly, getting the very unfortunate backtrace of:
    fuzzer_1  | Exception in <fuzzinator.job.validate_job.ValidateJob object at 0xf5183350>: decorator() missing 1 required positional argument: 'filename'
    fuzzer_1  | Traceback (most recent call last):
    fuzzer_1  |   File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/controller.py", line 413, in _run_job
    fuzzer_1  |     for issue in job.run():
    fuzzer_1  |   File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/job/validate_job.py", line 29, in run
    fuzzer_1  |     _, new_issues = self.validate()
    fuzzer_1  |   File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/job/validate_job.py", line 33, in validate
    fuzzer_1  |     sut_call, sut_call_kwargs = config_get_callable(self.config, 'sut.' + self.sut_name, ['validate_call', 'reduce_call', 'call'])
    fuzzer_1  |   File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/config.py", line 74, in config_get_callable
    fuzzer_1  |     entity = decorator(entity)
    fuzzer_1  |   File "/jscfuzz/venv/lib/python3.7/site-packages/fuzzinator/call/callable_decorator.py", line 32, in __call__
    fuzzer_1  |     return self.decorator(*self.decorator_args, **self.decorator_kwargs)(callable)
    fuzzer_1  | TypeError: decorator() missing 1 required positional argument: 'filename'
    This looks like I am missing filename for validation. But I have no idea why or where it's needed. Is there any way to find more information?
    Renáta Hodován
    @renatahodovan

    Can I access anything for which I defined in a property decorator?

    yes, you can access all of the fields of the issue dicts (including the result of property decorators, stdout, stderr, exit code, etc.)

    Paulo Matos
    @pmatos
    Wonder if you know the best way to deal with execution command line. The command is jsc {options} {test}
    Is there a way in jinja to format that string?
    Renáta Hodován
    @renatahodovan

    This looks like I am missing filename for validation. But I have no idea why or where it's needed. Is there any way to find more information?

    filename is probably missing from the file_writer_decorator:
    https://github.com/renatahodovan/fuzzinator/blob/master/fuzzinator/call/file_writer_decorator.py#L45

    Paulo Matos
    @pmatos
    Currently the report shows that string verbatim, so not very useful.
    Renáta Hodován
    @renatahodovan

    Wonder if you know the best way to deal with execution command line. The command is jsc {options} {test}

    yeah, you can use markdown formatting

    Paulo Matos
    @pmatos
    you mean, instead of jinja, are there any examples of that?
    Renáta Hodován
    @renatahodovan
    nope
    Paulo Matos
    @pmatos
    how would markdown help with formatting the string? Apparently I still need to use the {{ ... }} syntax.
    Renáta Hodován
    @renatahodovan
    I use it in case of jerry since its bugtracker is github which supports markdown, but not for webkit/jsc since bugzilla will ignore this formatting (AFAIK)
    so, where do you want to see the formatting? on the Fuzzinator issue page or on the bugtracker?
    Paulo Matos
    @pmatos
    Ah - I think I know what you mean. I didn't mean format the report with bold, etc. I meant that the string jsc {options} {test} needs to have the options, test variables interpolated with values from the issue.
    Renáta Hodován
    @renatahodovan
    aaahh
    Paulo Matos
    @pmatos
    I meant formatting in terms of 'jsc {options} {test}'.format(options=..., test=...) ...
    that's wrong syntax but i guess you know what i mean.
    Renáta Hodován
    @renatahodovan
    okay, that should be done automatically by fuzzinator if those fields (options, test) are present in your issue dictionary to be displayed
    Paulo Matos
    @pmatos
    humm, that's not happening, though. OK, I will investigate further. thanks.
    Renáta Hodován
    @renatahodovan
    if you change the url of your issue from
    localost:8080/issues/60d20f8d2fc48a39830ae7e7 to localhost:8080/api/issues/60d20f8d2fc48a39830ae7e7/ then you can see the encoded dict (the keys are readable)
    you should check if the referred fields are present