Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Harish Nataraj
    @harish.nataraj:matrix.org
    [m]

    :point_up: Edit: ## make_case is not transmitting my body in POST requests

    I have the below python code (also in the attachment above, and schema attached). However the case.call is not transmitting a body, and the content length is always 0. I even tried setting the content-length header, but to no avail.

    ##What am I doing wrong?

    import schemathesis
    import json
    import requests

    base_url = "http://35.226.144.72/"
    targeted_endpoint = "/identity/api/auth/login"

    schema = schemathesis.from_path("openapi.yaml",base_url=base_url)

    case = schema[targeted_endpoint]["POST"].make_case(body={"email":"loris@sysdig.com","password":"Bb123456"})
    response = case.call(headers={"Content-Type":"application/json"})
    print(response)

    3 replies
    Harish Nataraj
    @harish.nataraj:matrix.org
    [m]

    On another note:

    you mentioned that you have a PR that generates "negative" inputs for the API schema (that you are yet to merge). Is that PR for public review?
    5 replies
    Harish Nataraj
    @harish.nataraj:matrix.org
    [m]

    :point_up: Edit: ## make_case is not transmitting my body in POST requests

    I have the below python code (also in the attachment above, and schema attached). However the case.call is not transmitting a body, and the content length is always 0. I even tried setting the content-length header, but to no avail.

    ##What am I doing wrong?

    import schemathesis
    import json
    import requests

    base_url = "http://xx.xx.xx.xx/"
    targeted_endpoint = "/identity/api/auth/login"

    schema = schemathesis.from_path("openapi.yaml",base_url=base_url)

    case = schema[targeted_endpoint]["POST"].make_case(body={"email":"loris@sysdig.com","password":"Bb123456"})
    response = case.call(headers={"Content-Type":"application/json"})
    print(response)

    Harish Nataraj
    @harish.nataraj:matrix.org
    [m]
    How would you describe Schemathesis when compared to PACT? PACT also has schema conformance tests in addition to consumer driven contract tests via PACT broker.
    1 reply
    Buchi Reddy Busi Reddy
    @buchireddy
    @Stranger6667 Also, if we make some local changes to the Schemathesis CLI, how can I test and run locally? Are the instructions written anywhere? Sorry if I couldn't find them.
    6 replies
    Aprila Hijriyan
    @aprilahijriyan
    Hi @Stranger6667, does schemathesis have support for detecting endpoints that are protected with security schemes (eg oauth2 -> flows: password)
    1 reply
    jkhsjdhjs
    @jkhsjdhjs:totally.rip
    [m]

    Hey, I'm currently writing tests for a http api implemented with werkzeug. The API should be conformant to an OAS I wrote, that's why I decided to use schemathesis to automate the testing. Here's a link to a shortened version of the schema with some example code that should just illustrate how I'm using the schemathesis library: https://gist.github.com/jkhsjdhjs/51583738ac438eefd52f93a4723f2c4d

    The problem I have is that schemathesis (or rather hypothesis-jsonschema) generates invalid data for the schema, e.g. Case(body={'idShort': 'A', 'category': '', '': {}, '¤': [{"'": None}, {}, []]}). The idShort and category attributes are correct, but the schema doesn't have an empty attribute and neither a ¤ attribute. Since negative testing isn't implemented yet, I guess that's not what's going on here. I also found this issue (schemathesis/schemathesis#801), but this only regards empty attributes, not attribute names with crazy characters in them. My current step is to implement stateful testing, but since nearly all requests are rejected this makes it nearly impossible to test statefully.

    Maybe it's also my schema that's invalid, I'd be glad if someone could take a look. Thanks in advance!

    3 replies
    Alexander Efimov
    @loopguard
    Hey, first time trying to use schemathesis, what is the best way to specify valid query parameters when testing CRUD with Pytest? I tried using schema.add_link(), but seems like query parameter from the source is never applied to the target. What am I missing?
    8 replies
    Michael Nguyen
    @lexwraith_gitlab
    Hello, I'd like to take on schemathesis/schemathesis#921. It looks like it's related to https://github.com/schemathesis/schemathesis/blob/1050b81abdd6a51f6c67d35d5ec83758cda8a678/src/schemathesis/parameters.py#L85, where only the first example is explicitly used via the comments and code. Am I on the right track here? Tracing it down into the APIOperation project seems to somewhat validate what I'm thinking
    9 replies
    Also, is it worth adjusting this when schemathesis/schemathesis#1065 seems like the larger/cohesive project?
    Michael Nguyen
    @lexwraith_gitlab
    Also, side note, what exactly am I supposed to do to set up for contributing? It seems like installing pre-commit and using it will set me up with -most- things, but there are seemingly some PATHing issues when using tox when used with pyvenv, i.e. I'm using 3.7.9 systemwide but tox looks for 3.6 at some point, and tox also can't seem to find mypy et al. despite it being installed as part of pre-commit
    1 reply
    jkhsjdhjs
    @jkhsjdhjs:totally.rip
    [m]

    Hey, it's me again. Thanks for your help last time, I now rewrote the schema, replaced all allOf inheritance with YAML anchors and added additionalProperties: false to all definitions. Data generation is working great now!
    I'm currently testing stateful testing. The tests complete successfully, but I'm not sure if the API is actually tested correctly. I defined links in the components/links section of my OAS and referenced them at appropriate locations using $ref. To test this I intentionally broke the links by renaming a parameter to something non-existant:

          parameters:
            view-idShort: "$response.body#/idShort"

    In this case I removed the last character t of $response.body#/idShort such that it now references an attribute of the response object which doesn't exist. However, the tests still aren't failing and I don't know why. Do you have any idea what the problem could be?

    Another thing I'm wondering about: If a link is tested by schemathesis I want the next API call to return a 2xx status code, otherwise the tests should fail. Is this the default behaviour?

    6 replies
    Dmitry Dygalo
    @Stranger6667
    Hi all! I am gathering feedback on Schemathesis :) Please, feel this survey - https://forms.gle/arTf6yQEcQH8ShxHA (if you didn't do it already)
    It will help me to learn more about Schemathesis usage and decide on future development! :) It will take 5 minutes. The results are anonymous.
    Buchi Reddy Busi Reddy
    @buchireddy

    Hi @Stranger6667 this is a question about the below code in the serializers.py

    @register("multipart/form-data")
    class MultipartSerializer:
        def as_requests(self, context: SerializerContext, value: Any) -> Dict[str, Any]:
            if isinstance(value, bytes):
                return {"data": value}
            multipart = _prepare_form_data(value)
            files, data = context.case.operation.prepare_multipart(multipart)
            return {"files": files, "data": data}
    
        def as_werkzeug(self, context: SerializerContext, value: Any) -> Dict[str, Any]:
            return {"data": value}

    Why is the assumption that the value that comes in for multipart/form-data is either of type bytes or a dictionary? It could be a string type too right?

    10 replies
    Dhiraj Barnwal
    @djbarnwal
    Hi @Stranger6667 is there a standard way of allocating multiple worker threads in the Python version? I believe --workers solves the issue in the CLI version.
    4 replies
    Dhiraj Barnwal
    @djbarnwal
    Hey @Stranger6667 sorry to trouble you again. I have couple of more questions for you
    1. Is there a way to store network logs (as done in CLI) using pytest version? I believe just adding a mark decorator (using pytest-recording ) won't work directly with schemathesis?
    2. Is there a way to skip an endpoint while testing the schema?
    7 replies
    Dhiraj Barnwal
    @djbarnwal
    @Stranger6667 What's the status of schemathesis/schemathesis#947 ? Do you need any help with this PR?
    1 reply
    Vasco Veloso
    @vveloso
    Hi all, and @Stranger6667 in particular. :) My use case requires mutual TLS and I was looking at the documentation but could not understand whether it is possible to provide the client certificate and private key. Is this supported?
    3 replies
    Dhiraj Barnwal
    @djbarnwal

    @Stranger6667 Can schemathesis handle all type of characters? One of the API tests failed with this error

    self = <encodings.cp437.IncrementalEncoder object at 0x000002BF772ED520>, input = "{'³': None}", final = False
    
        def encode(self, input, final=False):
    >       return codecs.charmap_encode(input,self.errors,encoding_map)[0]
    E       UnicodeEncodeError: 'charmap' codec can't encode character '\xb3' in position 2: character maps to <undefined>
    
    c:\python39\lib\encodings\cp437.py:19: UnicodeEncodeError
    --------------------------------------------------------------------------------------------------------------- Hypothesis ----------------------------------------------------------------------------------------------------------------
    Falsifying example: test_api(
        case=Case(body={'³': None}),
    )

    Is Hypothesis producing examples which the test can't encode properly?

    3 replies
    sachin kn
    @sachinlionel
    Hi, Does anybody integrate schemathesis tests with robot framework? Are there are any example writing test case as function? I have seen pytest examples but I cannot use pytest runner.
    14 replies
    sachin kn
    @sachinlionel
    @Stranger6667 can you help me here? why do i see Unresolvable JSON pointer: 'components/schemas/geometryGeoJSON' ? this is the openapi spec in under test https://github.com/radiantearth/stac-api-spec/blob/master/item-search/openapi.yaml
    9 replies
    Ilya Prost
    @ilyasimas.prost_gitlab
    Hi, @Stranger6667, I'm trying to use schemathesis. I wrote easest example for python from docs, but it was more load testing cause api dropped under load. So I started looking for way to limit load. I found only max_example parameter, but it controls not RPS, it's for requests count, is there some way to limit load, some parameter for RPS control?
    1 reply
    johnnylayghton
    @johnnylayghton:matrix.org
    [m]
    Hi folks! I wanted to ask does Schemathesis also allows to create examples with post/put/patch requests that require a valid body? Similar how you do it with e.g. schema['/users/{id}/avatar']['get'].make_case(path_parameters={"id":1}). Basically I'm not interested in manually setting the whole body but just giving it the sufficient parameters for a valid example to be created (/cc @Stranger6667 😊)
    johnnylayghton
    @johnnylayghton:matrix.org
    [m]

    (to make the example more clear, say we are dealing with https://petstore3.swagger.io/ and POST /store/order):
    e.g. I can get a working example via get_strategies_from_examples().body - which is great, but doesn't exist always. But basically I'm looking to do is:
    case = schema['/store/order']['post']
    case.make_example()
    case.body
    {
    "id": int,
    "username": "string",
    "firstName": "string,
    "otherRequiredParameter" :"something"
    ......
    }
    so that I could just do case.call with {"id":1, "username":"john", ...} etc.

    This sample operation is +- simple but I think you get the idea

    And if I were to do case.call without e.g. a valid {id}, {username} -> similar to how it functions with get operations, I would expect also an exception to happen. Hope I'm getting my question across +- clearly ☹️
    johnnylayghton
    @johnnylayghton:matrix.org
    [m]
    (so the closest I got so far was by accessing and iterating over endpoint.definition.resolved['requestBody']['content']['application/json']['schema']['properties'][i] )
    1 reply
    johnjbrown
    @johnjbrown:matrix.org
    [m]

    Hello! I'm a technical writer. I'd been looking for a way to contract test an OpenAPI spec. And this project looks very, very close to exactly what I'm looking for. However, I haven't been able to get it to work, and I was wondering if someone could help.

    First, I should probably explain what I'm trying to do, because I'm not really a programmer and I may have misinterpreted.

    What I want to do is:

    1. For each endpoint in the OAS3 spec, generate a request based on the examples in the endpoint's requestBody schema.
    2. Send that example request to a remote sandbox URL.
    3. Discover whether the OAS3's schema and the API's accepted requests are in line depending on whether the response is 200 or not.

    Let me mention that I'd ideally like to do this via CLI params, without coding (or with minimal coding).
    Would this be possible? If not, a simple "no" is appreciated.

    If it is, I'll post my attempt to use schemathesis and the traceback in thread ↓. It seems like I'm having a problem with my example filtering (or maybe that's a symptom of something else).

    5 replies
    johnjbrown
    @johnjbrown:matrix.org
    [m]
    johnjbrown
    @johnjbrown:matrix.org
    [m]

    I tried to run this command. At first I was having some "null" data errors, so I just want to test one operation first. I am using <placeholders> for the auth header and URL

    1schemathesis run -O "create_entity" -H '<auth_key_value_pair>' --show-errors-tracebacks --base-url https://<v1-url>.co
    `

    I got the traceback that I attached above.

    Maybe this is really about hypothesis. I followed the traceback's link to https://hypothesis.readthedocs.io/en/latest/healthchecks.html.

    But the problem is that I'm not even sure what examples I'm supposed to filter--I was hoping that I could just use the schema examples.

    Obviously, I'm quite programming illiterate so there maybe some basic python stuff that I just need to know. But I wanted to ask, because if it is possible to accomplish my task without coding, I think this tool might be really useful for people who work with the OAS3 spec but don't code.

    johnjbrown
    @johnjbrown:matrix.org
    [m]

    It's very interesting information, thank you! I had actually wondered why all objects in my various oneOfs could be sent by the parser I'm using. Do you mind if I share your explanation text with an API writer's group? I'll credit you.

    And, unfortunately, I don't think an invalid OAS3 would work for my client's use case, so it seems like, I'd have to get a developer to write some python (or figure it out myself--which I should do, but maybe not for this project :-) ). Or, just wait for 3.1.0 adoption.

    However, I am personally interested in seeing how it would work, so if you don't mind, I'll send a simplified schema with the added propertyNames keyword. I think there are probably quite a few people like me, technical writers who maintain an OAS3 spec without any coding knowledge. I think schemathasis might be able to solve a problem they didn't realize they had.

    johnjbrown
    @johnjbrown:matrix.org
    [m]

    :point_up: Edit: It's very interesting information, thank you! I had actually wondered why all the objects linked in my various oneOf schema could be simultaneausly sent by the parser I'm using. Well, that explains it :-). Do you mind if I share your explanation text with an API writer's group? I'll credit you.

    And, unfortunately, I don't think an invalid OAS3 would work for my client's use case, so it seems like I'd have to get a developer to write some python (or figure it out myself--which I should do, but maybe not for this project :-) ). Or, just wait for 3.1.0 adoption.

    However, I am personally interested in seeing how it would work, so if you don't mind, I'll send a simplified schema with the added propertyNames keyword. I think there are probably quite a few people like me, technical writers who maintain an OAS3 spec without any coding knowledge. Schemathasis might be able to solve a problem they didn't realize they had.

    2 replies
    Pau Gui
    @Paugui_gitlab

    Hello o/ this tool is great ! It's help me to spot some bugs on my REST API. Thank you for that :D

    However, I stuck on this issue:

    hypothesis.errors.Flaky: Tests on this API operation produce unreliable results:
    Falsified on the first call but did not on a subsequent one

    For the context, I get that error on POST /tags endpoint that create a tag. I have validation rule that ensure the tag's label is unique.
    So this error is legit because for the same call, the first will pass, but the second will get a 422 error (because the label is taken at the first run).

    Must I change my openAPI spec to explain this, but how ? Or should add some param to schematesis CLI to avoid this kind of test ?

    7 replies
    Dmitry Dygalo
    @Stranger6667

    Hi folks! Schemathesis 3.8.0 was released recently!
    It includes the negative testing feature - Schemathesis can generate examples that do not fit the API schema🙂
    Please, take a look - it could be enabled with -D negative in CLI or like this for Python tests:

    import schemathesis
    from schemathesis import DataGenerationMethod
    
    schema = schemathesis.from_uri(
        "http://example.com/swagger.json",
        data_generation_methods=[DataGenerationMethod.negative]
    )
    
    
    @schema.parametrize()
    def test_api(case):
        case.call_and_validate()

    I'd be happy to hear your feedback on this feature🙂

    johnjbrown
    @johnjbrown:matrix.org
    [m]

    Yep, I shared your explanation about oneOf on the API writers channel. One other workaround suggested there would be to add additionalproperties: false. However, this works only if properties in the oneOf or anyOf subschemas are the only properties. It doesn't work if there are other properties alongside, as with my API. I suppose you knew this, but just posting in case the information helps anyone else.

    As expected, there was also some interest in the overall idea of contract testing.

    Umut Geyik
    @umutgyk95
    Hi all, I am very new in Schemathesis. First off all, thanks for this awesome tool. I have a question for you. In my OpenApi spec, I have an endpoint that starts event streams. In the server side, I response with chunkes which uses HTTP chunked encoding and send data with this. I run the schemathesis as state machine and tests never finish because of event streaming. How can I control the test on my stream endpoint during the connection (not before, not after) ? How can I terminate the connection when x times passed ?
    3 replies
    Ivan
    @EasyItBlog_twitter
    Hi folks! The project is great, thanks!
    Can anyone please help with the error I get?
    E   hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 0 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
    E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.
    5 replies
    Ivan
    @EasyItBlog_twitter
    I have string(type: uri) used in the schema. How do I override the URI generation used by Schemathesis? It generates non-existing ones.
    6 replies
    Ivan
    @EasyItBlog_twitter
    Any idea how to proceed with the following error? I checked the docs (https://schemathesis.readthedocs.io/en/stable/stateful.html?highlight=flaky#corner-cases) but it doesn't say what to do about it.
    >   @settings(verbosity=Verbosity.debug)
    E   hypothesis.errors.Flaky: Hypothesis found 2 distinct failures, but 1 of them exhibited some sort of flaky behaviour.
    3 replies
    Alexander Efimov
    @loopguard

    Hey!
    I have a noob question, should schema.given(data=st.data()) decorator work with pytest? I'm trying to use it over a test function, which, among other things, expects the result of my custom fixtures in the arguments. When executed, it crashes with an error missing 1 required positional argument: 'data'.

    Source: https://schemathesis.readthedocs.io/en/stable/python.html#using-additional-hypothesis-strategies

    6 replies
    Chan Lee
    @MrDiggles2

    Hi folks - is there a way to modify the response before validation when running through the CLI?

    The service I'm testing against wraps its responses in the following format: { data: <PAYLOAD>, meta: {...} }. For one reason or another, the Swagger docs for this service describe the schema of PAYLOAD only.

    I'd like to be able to unwrap the response before handing it off for validation. I've looked at the available hooks (and tested a few out) but none of them seem to be a good fit. Could someone point me in the direction of a solution or a workaround that might work?

    4 replies
    kevgliss
    @kevgliss
    Has anyone had issues with case.call_asgi() not respecting the URL in the "servers" section of the OpenAPI spec?
    2 replies
    Andrea Mugx
    @mugx-fc
    Hello everybody :wave:
    do you know how to instruct schemathesis to not follow 3xx redirections ?
    5 replies
    Ilya Prost
    @ilyasimas.prost_gitlab
    Hello guys, how can i exclude some endpoint branches? For example : '/hooks/'?
    1 reply
    David Thomason
    @dtom90

    Hello, I'm trying to speed up test execution with pytest-xdist, but I am not seeing any difference, in fact I'm seeing it take slightly longer.
    Without -n auto:

    % pytest test_simple.py --local-spec=True --api=staging
    ============================================ test session starts ============================================
    platform darwin -- Python 3.8.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
    rootdir: /Users/david/dev/hackathon/nylas-api-tests, configfile: pytest.ini
    plugins: subtests-0.5.0, schemathesis-3.9.7, xdist-2.3.0, hypothesis-6.14.0, forked-1.3.0
    collected 1 item
    
    test_simple.py ......
    
    ============================================ 1 passed in 40.34s =============================================

    With -n auto:

    % pytest test_simple.py --local-spec=True --api=staging -n auto
    ============================================ test session starts ============================================
    platform darwin -- Python 3.8.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
    rootdir: /Users/david/dev/hackathon/nylas-api-tests, configfile: pytest.ini
    plugins: subtests-0.5.0, schemathesis-3.9.7, xdist-2.3.0, hypothesis-6.14.0, forked-1.3.0
    gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1] / gw8 [1] / gw9 [1] / gw10 [1] / gw11 [1]
    ......                                                                                                                                                                                                          [100%]
    ============================================ 1 passed in 42.86s ============================================

    Is there a configuration that I'm missing to get speed improvements?

    2 replies
    zhangyue4107
    @zhangyue4107
    my check return false but case success ,plz help me
    
    @schemathesis.register_check
    def number_check(response, case):
        print(json.loads(get_response_payload(response)),response.status_code)
        if response.status_code == 200:
            data = json.loads(get_response_payload(response))
            assert data['un_read_num'] == 1,'some error'
            return True
    
        if response.status_code == 400:
            return False
        else:
            print('false')
            return False
    
    
    class TestApi(TestCase):
        # @pytest.mark.usefixtures("use_fix")
        @schema.parametrize()
        @given(case=schema['/edu/zone/system/message/unread/num']['GET'].as_strategy())
        @settings(max_examples=1, deadline=1000)
        def test_pets(self, case):
            response = case.call()
            logger.info('response:{}'.format(response._content.decode('utf-8')))
            case.validate_response(response, additional_checks=(number_check,))
    2 replies
    Vasco Veloso
    @vveloso

    Hi, I have one question about data generated during testing (schemathesis version is 3.9.7).

    For example, with the following (partial) spec:

          parameters:
            - in: path
              name: world
              type: string
              pattern: '[0-9a-zA-Z\s]'
              maxLength: 100
              description: 'The name of the person being greeted.'
              required: true
              x-example: 'John Doe'

    and the following Python test:

    import schemathesis
    
    schema = schemathesis.from_path(
        "../../web/src/main/openapi/specification.yaml",
        base_url="https://localhost:8086/"
    )
    
    @schema.parametrize()
    def test_api(case):
        case.headers = case.headers or {}
        case.headers["Host"] = 'my.host.com'
        case.call_and_validate(cert='../../conf/src/main/environments/local/certs/client_0.pem')

    most URLs are of the form:

    GET /myendpoint/11116p%F1%B0%B0%80%F1%80%A7%AE

    Does this mean that Hypothesis is just using random characters from [0, 255] for strings, or that it is using random Unicode characters, maybe UTF-16?

    3 replies
    zhangyue4107
    @zhangyue4107
    I would like to know more about the usage of hooks, including various examples,and what should i return for the different hooks in the docs. can somoone show it for me thank u
    4 replies
    Dhiraj Barnwal
    @djbarnwal
    Hi @Stranger6667 is there a way to store failed checks which are shown in log (such as The received response does not conform to the defined schema!) in a file? Are these failed checks exposed through some APIs which I can use?
    2 replies
    splinter64
    @splinter64:matrix.org
    [m]
    Hi folks! Is it possible to increase the number of threads / workers when using python interface? I see that option on https://schemathesis.readthedocs.io/en/stable/cli.html?highlight=threads#concurrent-testing for the CLI but not for python and it would be really useful
    also and unrelated: is it possible to import a swagger 2.0 specification in JSON format? (I believe no but wanted to confirm)