Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Brian Okken
    [Adam Parkin, Test Podcast] I dunno if that helps me answer my original question, but thanks for sharing that, super insightful.
    [David Kotschessa, Test Podcast] and yet I also get not wanting to go overboard and be hell bent on having a test for every.single.function
    [David Kotschessa, Test Podcast] sometimes a test that takes place inside the container you are hitting and runs a few of those functions - who knows what to call it - unit, component, integration? Might still serve a very good purpose
    [Brian Okken, Test Podcast] Wow. I don’t think I can disagree more.
    [Adam Parkin, Test Podcast] That’s really interesting, I’d be interested in hearing your thoughts as to why.
    [Brian Okken, Test Podcast] Code coverage using unit tests is trivial. Just write little tests for all of your code. Done.
    [Brian Okken, Test Podcast] But what I really want code coverage to tell me is what parts of the system are actually being used.
    [Brian Okken, Test Podcast] So coverage while running tests that hit API points or even system tests seem more effective for that purpose.
    Brian Okken
    [Brian Okken, Test Podcast] Your question seemed more like “how do I run coverage on code sitting on a different machine”.
    [Brian Okken, Test Podcast] The answer of “run it on the same machine, if you can” is still reasonable. I think. I actually don’t know if you can split coverage onto a different machine. But if you can spin up a local server for testing purposes, on the same machine as the tests, the integration tests still work.
    [Brian Okken, Test Podcast] YMMV, of course, depending on your architecture.

    [Adam Parkin, Test Podcast] Unfortunately in my specific case here that (running the server locally alongside the test) is not an option (or at least not without a lot of work to make that happen, which maybe that’s the answer :shrug).

    But yes, effectively it is “test is running on one machine, code under test is running on a different machine”.

    [Adam Parkin, Test Podcast] And so my dilemma is I want to gain insight into which parts of the overall system (which is made up of many services/APIs) are covered/exercised by the test suite and which are not (so as to find gaps that need addressing).
    Brian Okken

    [Adam Parkin, Test Podcast] It occurs to me I’m probably misusing the term “integration test” here, in my case my tests are really “full system tests”, so maybe to tweak my question slightly: is it a good idea to generate coverage information for tests that are full system tests, and if so (or at the very least if it’s not a bad idea) how do you achieve that?

    Like maybe a more concrete way of thinking of this: say you have a bunch of selenium tests that you execute against your staging environment. You want to get a sense of what parts of the overall system are exercised by those tests and which are not, how would you go about discovering that?

    Brian Okken
    [David Kotschessa, Test Podcast] Yes @brian I know you are not a fan of the traditional "unit test" and the pyramid and all that, and I'm totallt sympathetic to your way of looking at it. That's why I followed up with my second statement, but I'm not sure where else to go from there in the context of the architecture in question.
    [Brian Okken, Test Podcast] I feel I need to preface my opinion with “I don’t do a lot of testing of web services”.
    [Brian Okken, Test Podcast] But I do test complex systems.
    Brian Okken
    [Brian Okken, Test Podcast] At some level there is a thing that gets deployed that is larger than a single function but smaller than “the system”. These things have interfaces, and seem like they should be testable as components from their interface. I think that’s the sweet spot of testing complex systems. Test all the parts (services?) with feature testing. I really like the feature testing article on the twitter dev blog, https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html
    [Brian Okken, Test Podcast] I’m not opposed to unit tests. But they are for developer sanity checking, in my use of them. They don’t tell me if the system works.
    [Brian Okken, Test Podcast] Please correct me if I’m wrong in the assumption that feature testing each service could use coverage effectively, and adding to that system level workflow tests would sufficiently test a system. Even a complex web API level system.
    [Brian Okken, Test Podcast] I personally love unit tests. But I’m also a classicist, and I think lots of people teaching TDD and testing are mockists. It’s the mockist approach to unit testing I’m opposed to. I’m also opposed to having everything fully unit tested by a dev team, firing QA, and having just the customers test the system.
    Brian Okken
    [Brian Okken, Test Podcast] The question of “can I run tests on one computer and collect coverage against code running on a different server” would be a great question for Ned Batchelder or even to the “testing in python” mailing list.
    Brian Okken

    [David Kotschessa, Test Podcast] Say I want to do something crazy with tox like run...well... like every version of python since 2.7. (It's an experiment and possibly blog article).

    I'm confused bout the overlapping domains of:
    virtual environments (I use just venv)
    tox itself (which, I guess is lke venv, but you can still use it with venv?)
    Now I'm reading i might want pyenv if I want to install all these different versions of python.
    What's the simplest way?

    Brian Okken
    [Tibor Arpas, Test Podcast] tox doesn’t install python versions. Also you have to specify libraries in the environments in tox.ini so you’ll not reuse the way you install them normally as a user from command line. tox is a lifesaver if you need a matrix of different dependency sets created and execute a (test) command in them. If you want to run the same thing for many different python versions it might not be the best tool.
    Brian Okken
    [David Kotschessa, Test Podcast] If you want to run the same thing for many different python versions it might not be the best tool.
    Is that not considered one if it's primary uses?
    Brian Okken
    [Tibor Arpas, Test Podcast] :) kind of. Maybe tell us more about your use case. Do you mean 12 versions of python 2.7 - 3.10? Are you sure a quick bash/python script wouldn’t be easiest?
    [Tibor Arpas, Test Podcast] tox docs says it uses virtualenv to create and manage many virtualenvs. I use tox with many different pythons versions and didn’t need pyenv in that context.
    [David Kotschessa, Test Podcast] It's not a real world use case. It's more of a giant experiment and learning experience involving writing code and watching it pass/fail in different versions to learn about the breaking changes and new features that came with each one.
    [David Kotschessa, Test Podcast] idea for an article
    [Tibor Arpas, Test Podcast] so, venv and tox overlap a little but I think easiest is to use tox (with it’s standard way to create virtualenvs and install dependencies)
    Brian Okken

    [David Kotschessa, Test Podcast] So I guess here's where I'm confused. say I'm in an activated venv and I install a package, say django - it installs itself in the venv/bin/whatever folder, but not otherwise on my machine.
    The python version I'm using is also in venv/bin/python (or whatever) based on what version I'm using. Bu it's also installed globally.

    So I have python 2.7 (because mac still ships with it) and 3.8 (because that's what I installed)
    Soooo say I want to install python 3.2 - is there an installation method that puts it in the venv but does not install globally?

    [David Kotschessa, Test Podcast] because I'm only going to use it for that project.
    [Tibor Arpas, Test Podcast] tox creates all the configured virtual environments under .tox directory and I think by default doesn’t take anything from global/active virtualenv.
    [Tibor Arpas, Test Podcast] i think it’s fine to have tox globally… and then all the things which you want experiment with are configured in tox.ini and corresponding virtualenv are independent from global install, from pyenv or venv
    Brian Okken
    [David Kotschessa, Test Podcast] oh yes I'm fine with having tox globally
    [David Kotschessa, Test Podcast] but I mean the different python versions
    [David Kotschessa, Test Podcast] Because python isn't 'package' but it still ends up in your environment. I can't pip install python3.2 or something (right?)
    [Tibor Arpas, Test Podcast] tox was able to find the base python versions just fine on my mac… if that’s not the case for you I think you’ll be able to help it … via config
    [David Kotschessa, Test Podcast] i'm not worried it will find them - but just wondering if there's an ideal way to install them
    [Tibor Arpas, Test Podcast] I installed everything “globally” distinguished via python3.5 python3.6 etc
    [David Kotschessa, Test Podcast] I guess it doesn't matter that they are globally installed
    [Tibor Arpas, Test Podcast] I think you can start in the easiest way and then change if needed. There is too many options and details and I think those will be more important than a “strategy”
    [David Kotschessa, Test Podcast] It's also that I have the intention of making this easy for someone else to follow.
    [Tibor Arpas, Test Podcast] everything is configured in tox.ini and it doesn’t interfere with virtualenvs not created with tox. The python versions - easiest to have them all on path. I’m sure there is a other way but I don’t have experience with that.
    Brian Okken
    [Tibor Arpas, Test Podcast] > It’s also that I have the intention of making this easy for someone else to follow.
    Understood. tox doesn’t install python versions and I don’t have experience with any fancy tool that would do that.
    [David Kotschessa, Test Podcast] No problem. You clarified a lot there for me though
    Brian Okken
    [David Kotschessa, Test Podcast] well, there's this... https://hub.docker.com/r/fkrull/multi-python
    [David Kotschessa, Test Podcast] kind of taking it up a notch
    [David Kotschessa, Test Podcast] I kind of wanted to write something somewhat beginner friendly but I'm now thinking with tox and multiple python versions I'm already leaving that arenea
    Brian Okken
    [Tibor Arpas, Test Podcast] nice :)
    Yeah tox is not very intuitive so not beginner friendly..