Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    olivia2046
    @olivia2046
    Hello Dmitry, schemathesis is very cool, yet I have one problem as the APIs I want to test are defined as the following example in swagger: GET /api/v{version}/Category/filter. {version} is the requested API version and is set as 1 in all APIs. So it should not be handled as other parameters(which will be tested against random inputs generated). Is there a way to set value to specific parameter in schemathesis?
    Dmitry Dygalo
    @Stranger6667

    Hello Olivia! Thank you :)
    At the moment it could be done by modifying call.path_parameters:

    def test_api(case):
        case.path_parameters["version"] = "1"
        # then usual `case.call` will contain the proper path
        response = case.call()

    The downside of it is slightly slower performance and variance than it could be. We have this issue in our issue tracker to address such cases - kiwicom/schemathesis#8
    I will increase the priority of this issue, so we can handle relevant cases easier and in a more performant way :)

    Let me know if this approach works for you :)
    olivia2046
    @olivia2046
    yeah, it works for me. Thanks a lot!
    Dmitry Dygalo
    @Stranger6667
    Great to hear it, you are welcome :) Feel free to reach us in case if you need any support on Schemathesis :)
    Yury Danilin
    @ydanilin
    Hello! Thank you for a good package! When generating request BODY payloads, tests include extra parameters which are not defined in the corresponding (OpenApi3) schema. Can I disable this behavior somehow (generate only declared parameters)?
    Dmitry Dygalo
    @Stranger6667
    Hello Yury! It could be done on the schema side by adding "additionalProperties": false to the relevant object definition, but this behavior uncovered some bugs in our apps - e.g. 3rd party folks were sending a field name with a typo and it was ignored, and some cases when typos in fields will crash our apps, since the payload was still fitting to the schema. However, if not declared attributes are ok or changing schema is not desired we can think about some hooks that will provide a way to change data generation behavior. For example it could be a hook, that will be able to modify schema / strategies in runtime, before a test session start. For example after "additionalProperties": false could be added in runtime, without modifying the source schema . What do you think?
    Dmitry Dygalo
    @Stranger6667
    And, by the way, thank you for using Schemathesis :)
    Yury Danilin
    @ydanilin

    Hi Dmitry!
    Thank you, "additionalProperties": false works!!

    The background of my question was that my framework (fastapi) allows to restrict JSON body values to only defined in the schema, so it always validates and returns 4xx "Extra field not allowed" to Schemathesis tests, they were useless.

    The reason I do not like to allow non-defined fields is evident - to allow users to see they do something wrong as early as possible - typos in names, casing issues, params which seemed domain-related but unsupported, etc.

    I think this could be a Schemathesis adjustable setting to allow or deny non-defined fields in tests and,
    if adding "additionalProperties": false in runtime is an easiest way, why not...

    Dmitry Dygalo
    @Stranger6667

    Aha, I didn't know that fastapi does this validation :) Cool! :)

    in general, I think that even if the framework has this validation, it is nice to have the schema also in line with this behavior - for example, 3rd parties could use this schema for validating their payload, so it will be safer to add "additionalProperties": false to the schema itself.

    But, nevertheless, I'll work on some hooks for adjusting schemas in runtime after concurrent tests execution for CLI :)

    Thank you, for bringing this issue :)
    James Cooke
    @jamescooke
    Hi hi :wave:
    Thanks for Schemathesis! I only started using it yesterday and already found a bug in Django Rest Framework as a result (https://github.com/encode/django-rest-framework/issues/7134#issuecomment-572990297)
    Dmitry Dygalo
    @Stranger6667
    Hej James :) Cool, thanks for using our project :) Let us know if we can somehow improve your experience with it
    James Cooke
    @jamescooke
    Hi Dmitry - it's working really well for me at the moment :+1:
    I've got a question about this output I'm getting: "nreliable test timings! On an initial run, this test took 797.46ms, which exceeded the deadline of 500.00ms, but on a subsequent run it took 166.09 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None." --- is there any way to set deadline=None via the command line invocation?
    I get Error: no such option: --deadline when I just stick --deadline=None on the end of the command
    James Cooke
    @jamescooke
    Solved my own problem - I found that the help on the run command gives the info: schemathesis run --help (not just schemathesis --help as in the README)
    Dmitry Dygalo
    @Stranger6667
    To clarify for the future readers :) Option to use None value in --hypothesis-deadline CLI argument was added in Schemathesis 0.22.0
    Bill Huneke
    @wahuneke

    Hi Dmitry. Awesome plugin! Really interested to be checking it out. Just getting started. I'm trying to use it to check my api which has a large swagger schema already. curious to see what it is actually testing, so I created a test to just print case examples. What I found is that most test cases look to be 100% identical. That is, something about my config causes schemathesis to parametrize to the same combination of request args.

    If things are working, shouldnt I tend to see different values for request args in every case?

    Thanks again.

    Dmitry Dygalo
    @Stranger6667

    Hello Bill! Thank you :)
    There could be multiple causes of such behavior.
    First of all, I need to say that data generation is sensitive to the input schema. Some schemas, especially without oneOf and nullable fields (from my experience on running schemathesis against our internal services), have very few duplicates in generated data, but some schemas have a lot of them.
    The input schema is handled by hypothesis-jsonschema package, which creates a Hypothesis strategy that generates data that fits the input schema.
    hypothesis-jsonschema is not feature-complete yet and therefore not optimized. Most efforts are put into having an acceptable average case and avoiding pathologically poor performance for certain schemas. A quote from Zac Hatfield-Doods who develops hypothesis-jsonschema:

    ... once mature (ie feature-complete) we will definitely want to address these problems.

    Zac-HD/hypothesis-jsonschema#24 (the quote is from the original issue description)
    Besides it, there are cases when Hypothesis itself generates duplicate samples:

    • Testing against flaky tests. E.g. the test case failed, then Hypothesis re-runs it to detect if it is a random failure or not
    • Some duplicated sub-strategies are not always deduplicated. It is a known performance issue in Hypothesis, more info - HypothesisWorks/hypothesis#2087 and HypothesisWorks/hypothesis#1864
      The situation changes over time and during the last few releases Hypothesis improved its performance, hopefully, the issue above will be solved as well in the near future.
      I have a couple of suggestions that may improve your case:
    • upgrade to the latest versions of hypothesis and hypothesis-jsonschema (if you do not have them installed already)
    • set a higher number for max_examples setting. By default it is 100, I usually try 1000 (and some bugs on our prod services were discovered with ~10000 examples). To do so you need to wrap your test with @hypothesis.settings(max_examples=1000) decorator or use --hypothesis-max-examples=1000 CLI option if you run Schemathesis in the command line.

    Maybe there is a case in your schema that is not covered in hypothesis-jsonschema - it will be helpful if you'll share some minimal example of a schema that have a lot of duplicated values.
    Hope this helps :)

    missial
    @MissiaL

    @Stranger6667 HI! Thanks for good plugin!

    I try to disable some checks , like that

    def filter_chars(strategy):
        chars = ('\\x00', "\\ud")
        _filter = lambda value: not (set(value) & set(chars))
        return strategy.filter(_filter)
    
    schemathesis.hooks.register("body", filter_chars)
    schemathesis.hooks.register("query", filter_chars)
    schemathesis.hooks.register("form_data", filter_chars)
    schemathesis.hooks.register("path_parameters", filter_chars)
    schemathesis.hooks.register("headers", filter_chars)

    But this does not help and my application continues to catch such characters. Tell me what am I doing wrong?

    missial
    @MissiaL
    Fixed. My solution is
    def install_hooks():
        def filter_chars(strategy):
            def _filter_chars(value):
                chars = ('\\x00', '\\ud')
                return not any(c for c in chars if c in str(value))
    
            return strategy.filter(_filter_chars)
    
        for hook in HookLocation.__members__:
            schemathesis.hooks.register(hook, filter_chars)
    
    
    install_hooks()
    Dmitry Dygalo
    @Stranger6667

    Hello @MissiaL ! You're very welcome :)
    You have "\\ud" which is only a part of a Unicode char double-escaped representation, the actual chars are different in a set:

    In [1]: target = "\ud800a\ud900"
    
    In [2]: set(target)
    Out[2]: {'a', '\ud800', '\ud900'}
    
    In [3]: "\\ud" in set(target)
    Out[3]: False

    Also, the filter might receive values of different types, dictionaries most often, and after calling set in them there will be only keys remaining, and values are not checked:

    In [8]: set({"a": target})
    Out[8]: {'a'}

    but when str is called on a dict with Unicode chars, they become escaped:

    In [9]: str({"a": target})
    Out[9]: "{'a': '\\ud800a\\ud900'}"

    and therefore \\ud is found in it. Converting to strings is the simplest solution that can be - an alternative is traversing the generated value (it may be highly nested structures) and checking keys and values for being a string and then check if they contain the unwanted chars (with a regexp on unicode ranges most probably will be the easiest option)

    Hope this helps, I think we need to add more details to our docs on this + maybe add a helper to install one filter on all places

    ludwig404
    @leonh

    Just trying out schemathesis seems very good so far .....is there a way to list what calls have been made to my api?
    running with

    schemathesis run --validate-schema False --hypothesis-verbosity normal --workers 10 --hypothesis-report-multiple-bugs False --request-timeout 100 http://0.0.0.0:8082/openapi.json

    platform Darwin -- Python 3.8.1, schemathesis-0.24.3, hypothesis-5.5.4, hypothesis_jsonschema-0.11.1, jsonschema-3.2.0
    rootdir: /Users/leon/envs/fastapi-quickstart
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/leon/envs/fastapi-quickstart/.hypothesis/examples')
    Schema location: http://0.0.0.0:8082/openapi.json
    Base URL: http://0.0.0.0:8082
    Specification version: Open API 3.0.2
    Workers: 10
    collected endpoints: 11

    ..E..EF....

    summary

    not_a_server_error 609 / 609 passed PASSED

    Origin1227
    @Origin1227
    Hi am new to schemathesis, i tried its working great. btw Is there anyway to to check urls along with query params?
    Dmitry Dygalo
    @Stranger6667

    Just trying out schemathesis seems very good so far .....is there a way to list what calls have been made to my api?

    Hello @leonh :) Thanks!
    At the moment Schemathesis shows only minimal payload examples on errors, but we can add some --verbosity option to the CLI and display all requests / responses in a verbose mode.
    Added an issue for it - kiwicom/schemathesis#423

    Hi am new to schemathesis, i tried its working great. btw Is there anyway to to check urls along with query params?

    Hi @Origin1227 ! Nice to hear that :) Do you mean generation of some urls and check if they are handled correctly? Could you please, elaborate? Some example would be great

    Origin1227
    @Origin1227

    @Stranger6667 am looking for generation of urls along with query params
    eg: /articles?include="abc"
    i want to add a custom check for response for the urls with query params
    i tried this this with pytest

       @schema.parametrize()
       def test_include_param(client,case):
          url = "http://127.0.0.1:4010" + case.formatted_path            
                  response = client.request(
                    case.method,
                    url,
                    params=case.query,
                    json=case.body
                )
    #CHECKING RESPONSE._CONTENT

    but trying it with custom checks, case.query is always None how can i include query params in the case

       @schemathesis.register_check
       def test_include_pram(response, case):

    it would be great if you could clarify some more things

    • is there a way to export the test result in junit.xml format
    • is the response of failed tests stored somewhere
    ludwig404
    @leonh
    @Stranger6667 thanks for the reply, I noticed that some requests being made that really slowed my api down was trying to pinpoint which ones. The random way of testing is interesting and highlights things you wouldn't normally think of.
    Dmitry Dygalo
    @Stranger6667

    @Origin1227

    but trying it with custom checks, case.query is always None how can i include query params in the case

    It depends on your schema, at the moment Schemathesis only generates data, that fits your schema, as the next step we plan to introduce generating of data that doesn't fit the schema + later generation of random URLs and parameters that do not exist in the particular schema at all. For example, there is only a body with a single field defined in the schema (let say {"id": {"type": "integer"}}) and no additional fields are allowed, in this case, Schemathesis currently:

    • will generate bodies that fit into the definition (e.g {"id": 1} or {} if "id" is optional);
    • will NOT generate some other types for "id", i.e {"id": "test"} will not be generated;
    • will NOT generate "query" attributes since it is not defined;
      We're going to implement two latter cases in the future to improve variety of test cases. Also, it is possible to craft such a test manually for a particular endpoint:
      @given(
        case=schema["/endpoint"]["get"].as_strategy(),
        query=st.dictionaries(st.text(min_size=1, max_size=5), st.integers())  # or any other `dictionaries` strategy
      )
      def test_include_param(case, query):
        ...

    is there a way to export the test result in junit.xml format

    The pytest built-in plugin should work:
    pytest --junitxml=report.xml tests

    is the response of failed tests stored somewhere

    Not yet, but there is an issue for it - kiwicom/schemathesis#379

    Cheers

    Origin1227
    @Origin1227
    thanks @Stranger6667 :) , btw i want to export the results in junit.xml for checks like Response schema conformance etc.. which am running through shemathesis cli is there a way to do that?
    Dmitry Dygalo
    @Stranger6667
    @Origin1227 oh, I didn't think about such feature :) kiwicom/schemathesis#427 - will add it to our backlog
    Origin1227
    @Origin1227
    ty @Stranger6667
    btw found a issue when in schema there is a ref from A to B abd b also contains a ref B to A
    RecursionError: maximum recursion depth exceeded while calling a Python object
      {
        article  : {
           abc : "$ref":"author"
        },
        author : {
           efg : "$ref":"article "
        }
      }
    Dmitry Dygalo
    @Stranger6667
    Yep, recursive references are not supported:(