Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Brian Okken
    @okken
    [Brian Okken, Test Podcast] So coverage while running tests that hit API points or even system tests seem more effective for that purpose.
    [Brian Okken, Test Podcast] Your question seemed more like “how do I run coverage on code sitting on a different machine”.
    [Brian Okken, Test Podcast] The answer of “run it on the same machine, if you can” is still reasonable. I think. I actually don’t know if you can split coverage onto a different machine. But if you can spin up a local server for testing purposes, on the same machine as the tests, the integration tests still work.
    [Brian Okken, Test Podcast] YMMV, of course, depending on your architecture.

    [Adam Parkin, Test Podcast] Unfortunately in my specific case here that (running the server locally alongside the test) is not an option (or at least not without a lot of work to make that happen, which maybe that’s the answer :shrug).

    But yes, effectively it is “test is running on one machine, code under test is running on a different machine”.

    Brian Okken
    @okken
    [Adam Parkin, Test Podcast] And so my dilemma is I want to gain insight into which parts of the overall system (which is made up of many services/APIs) are covered/exercised by the test suite and which are not (so as to find gaps that need addressing).
    Brian Okken
    @okken

    [Adam Parkin, Test Podcast] It occurs to me I’m probably misusing the term “integration test” here, in my case my tests are really “full system tests”, so maybe to tweak my question slightly: is it a good idea to generate coverage information for tests that are full system tests, and if so (or at the very least if it’s not a bad idea) how do you achieve that?

    Like maybe a more concrete way of thinking of this: say you have a bunch of selenium tests that you execute against your staging environment. You want to get a sense of what parts of the overall system are exercised by those tests and which are not, how would you go about discovering that?

    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] Yes @brian I know you are not a fan of the traditional "unit test" and the pyramid and all that, and I'm totallt sympathetic to your way of looking at it. That's why I followed up with my second statement, but I'm not sure where else to go from there in the context of the architecture in question.
    [Brian Okken, Test Podcast] I feel I need to preface my opinion with “I don’t do a lot of testing of web services”.
    [Brian Okken, Test Podcast] But I do test complex systems.
    Brian Okken
    @okken
    [Brian Okken, Test Podcast] At some level there is a thing that gets deployed that is larger than a single function but smaller than “the system”. These things have interfaces, and seem like they should be testable as components from their interface. I think that’s the sweet spot of testing complex systems. Test all the parts (services?) with feature testing. I really like the feature testing article on the twitter dev blog, https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html
    [Brian Okken, Test Podcast] I’m not opposed to unit tests. But they are for developer sanity checking, in my use of them. They don’t tell me if the system works.
    [Brian Okken, Test Podcast] Please correct me if I’m wrong in the assumption that feature testing each service could use coverage effectively, and adding to that system level workflow tests would sufficiently test a system. Even a complex web API level system.
    [Brian Okken, Test Podcast] I personally love unit tests. But I’m also a classicist, and I think lots of people teaching TDD and testing are mockists. It’s the mockist approach to unit testing I’m opposed to. I’m also opposed to having everything fully unit tested by a dev team, firing QA, and having just the customers test the system.
    Brian Okken
    @okken
    [Brian Okken, Test Podcast] The question of “can I run tests on one computer and collect coverage against code running on a different server” would be a great question for Ned Batchelder or even to the “testing in python” mailing list.
    Brian Okken
    @okken

    [David Kotschessa, Test Podcast] Say I want to do something crazy with tox like run...well... like every version of python since 2.7. (It's an experiment and possibly blog article).

    I'm confused bout the overlapping domains of:
    virtual environments (I use just venv)
    tox itself (which, I guess is lke venv, but you can still use it with venv?)
    Now I'm reading i might want pyenv if I want to install all these different versions of python.
    What's the simplest way?

    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] tox doesn’t install python versions. Also you have to specify libraries in the environments in tox.ini so you’ll not reuse the way you install them normally as a user from command line. tox is a lifesaver if you need a matrix of different dependency sets created and execute a (test) command in them. If you want to run the same thing for many different python versions it might not be the best tool.
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] If you want to run the same thing for many different python versions it might not be the best tool.
    Is that not considered one if it's primary uses?
    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] :) kind of. Maybe tell us more about your use case. Do you mean 12 versions of python 2.7 - 3.10? Are you sure a quick bash/python script wouldn’t be easiest?
    [Tibor Arpas, Test Podcast] tox docs says it uses virtualenv to create and manage many virtualenvs. I use tox with many different pythons versions and didn’t need pyenv in that context.
    [David Kotschessa, Test Podcast] It's not a real world use case. It's more of a giant experiment and learning experience involving writing code and watching it pass/fail in different versions to learn about the breaking changes and new features that came with each one.
    [David Kotschessa, Test Podcast] idea for an article
    [Tibor Arpas, Test Podcast] so, venv and tox overlap a little but I think easiest is to use tox (with it’s standard way to create virtualenvs and install dependencies)
    Brian Okken
    @okken

    [David Kotschessa, Test Podcast] So I guess here's where I'm confused. say I'm in an activated venv and I install a package, say django - it installs itself in the venv/bin/whatever folder, but not otherwise on my machine.
    The python version I'm using is also in venv/bin/python (or whatever) based on what version I'm using. Bu it's also installed globally.

    So I have python 2.7 (because mac still ships with it) and 3.8 (because that's what I installed)
    Soooo say I want to install python 3.2 - is there an installation method that puts it in the venv but does not install globally?

    [David Kotschessa, Test Podcast] because I'm only going to use it for that project.
    [Tibor Arpas, Test Podcast] tox creates all the configured virtual environments under .tox directory and I think by default doesn’t take anything from global/active virtualenv.
    [Tibor Arpas, Test Podcast] i think it’s fine to have tox globally… and then all the things which you want experiment with are configured in tox.ini and corresponding virtualenv are independent from global install, from pyenv or venv
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] oh yes I'm fine with having tox globally
    [David Kotschessa, Test Podcast] but I mean the different python versions
    [David Kotschessa, Test Podcast] Because python isn't 'package' but it still ends up in your environment. I can't pip install python3.2 or something (right?)
    [Tibor Arpas, Test Podcast] tox was able to find the base python versions just fine on my mac… if that’s not the case for you I think you’ll be able to help it … via config
    [David Kotschessa, Test Podcast] i'm not worried it will find them - but just wondering if there's an ideal way to install them
    [Tibor Arpas, Test Podcast] I installed everything “globally” distinguished via python3.5 python3.6 etc
    [David Kotschessa, Test Podcast] I guess it doesn't matter that they are globally installed
    [Tibor Arpas, Test Podcast] I think you can start in the easiest way and then change if needed. There is too many options and details and I think those will be more important than a “strategy”
    [David Kotschessa, Test Podcast] It's also that I have the intention of making this easy for someone else to follow.
    [Tibor Arpas, Test Podcast] everything is configured in tox.ini and it doesn’t interfere with virtualenvs not created with tox. The python versions - easiest to have them all on path. I’m sure there is a other way but I don’t have experience with that.
    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] > It’s also that I have the intention of making this easy for someone else to follow.
    Understood. tox doesn’t install python versions and I don’t have experience with any fancy tool that would do that.
    [David Kotschessa, Test Podcast] No problem. You clarified a lot there for me though
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] well, there's this... https://hub.docker.com/r/fkrull/multi-python
    [David Kotschessa, Test Podcast] kind of taking it up a notch
    [David Kotschessa, Test Podcast] I kind of wanted to write something somewhat beginner friendly but I'm now thinking with tox and multiple python versions I'm already leaving that arenea
    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] nice :)
    Yeah tox is not very intuitive so not beginner friendly..
    Brian Okken
    @okken

    [Gabriele Bonetti, Test Podcast] Hi All, currently I run some test pipelines on jenkins completely on AWS, creating on-demand workers (jenkins slaves) when they are needed. Workers have been linux only so far but we will introduce win 10 support which needs to be tested. Have you ever tried to manage win 10 images on AWS? since there is no native image I see you have to bring your own and manage your own licenses, but looks cumbersome, how it's going to work on on-demand dynamic workers that then disappear?

    So I was wondering if this is a good idea at all or there are other solutions, for example using the supported windows server 2016 / 2019 images with desktop support and that can be good enough for testing on the equivalente windows desktop.

    For example I was also looking at browerstack, and there is a server -> desktop table, except for windows 10 :-) https://www.browserstack.com/question/478#:~:text=Home%20Support%20Live-,Is%20web%20testing%20on%20Windows%20Server%20edition%20the%20same%20as,remote%20access%20to%20our%20customers.&text=For%20Windows%2010%2C%20we%20use,which%20is%20a%20Desktop%20edition.

    Brian Okken
    @okken

    [Kristoffer Bakkejord, Test Podcast] Anyone have experience with this course? https://www.udemy.com/course/elegant-automation-frameworks-with-python-and-pytest/

    I would like to improve the pytest knowledge in my team, and addressing it through a not-too-long video course seems like it could be a good idea.

    [Kristoffer Bakkejord, Test Podcast] Or is there any other introduction video courses you recommend?
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] No but No but that reminds me @brian really needs to turn his book into a course. You still have any aspirations on that Brian?
    Brian Okken
    @okken

    [Pax, Test Podcast] Hello! Anyone here going to PyCon US next week? I’m looking for someone to replace my spot as volunteer for the Sponsor Workshop to this year’s PyCon US, “Blackfire: Debugging Performance in Python and Django applications - (Jérôme Vieilledent, Sümer Cip)”, which is happening on (Eastern US) 15:00 to 16:30 on Thursday May 13.

    I will give more info (what the role generally entails) here if anyone is a bit interested. I’m hesitant to send a super long message especially when I haven’t posted in a while on here. :see_no_evil:

    Brian Okken
    @okken

    [Jacob Floyd, Test Podcast] Some pytest help please:
    Is there a way to access capsys in a unittest-based test class without preventing nosetest from running the test as well? (only use capsys if running under pytest, else continue with the test's current stdout/stderr redirection logic)

    I've got a unittest based test suite that I'm working on running under pytest. I'm trying to minimize the delta of my changes to avoid integration hell as my branch won't be merged until all tests are working under the new setup (more than just pytest). In the meantime the tests will continue to run under nosetest, and I will need to periodically merge in the changes from the main branch into my pytest branch (thus, minimizing deltas is really important).

    I have one TestCase class that replaces sys.stdout and sys.stderr with open(path_to_temp_file, "w") in setUp() and then sets sys.stdout = sys.__stdout__ in tearDown(). That path_to_temp_file is then inspected in the tests to make sure output of a command isn't broken.
    This, of course conflicts with pytest's output capturing. And so the test seems to hang for a couple minutes before passing.
    In a pure pytest test, I would reach for the capsys fixture. But is there a way to use that in a unittest-based test? Hopefully conditionally based on if running under pytest or not.

    Brian Okken
    @okken
    [Jacob Floyd, Test Podcast] or is there a mark I can use to disable pytest stdout/stderr capturing for a class?