Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] Looks like I forgot to add this slack back in when I got a new computer. How's everybody doing? Seems quiet!
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] This is where I get up on stage and scream " I CAN'T HEAR YOU!!"
    [David Kotschessa, Test Podcast] https://www.youtube.com/watch?v=C1E10PRfhcA
    [David Kotschessa, Test Podcast] =D
    Brian Okken
    @okken

    [David Kotschessa, Test Podcast] Thank you @brian for an episode back in 2019 "From python script to maintainable package."

    I created my very first pip installable package, which is a community provider for faker to create airport data. It's a tiny thing, but it was a great lesson in documentation, setting up tests, packaging, etc.
    https://pypi.org/project/faker_airtravel/

    [David Kotschessa, Test Podcast] My excitement is very disproportional to how big or useful the package is. but it's the first thing I've ever put in pypi
    [David Kotschessa, Test Podcast] flit was definitely the way to go too
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] going through some of the other motions now, tox etc.
    Brian Okken
    @okken
    [AJ Kerrigan, Test Podcast] If this affects you you've probably already been notified, but... https://about.codecov.io/security-update/
    Brian Okken
    @okken

    [Tony Cappellini, Test Podcast] How do you deal with python3’s end= in the print() function, when using the python logger?

    print(‘Python Rocks’, end=‘’)

    print(‘Python Rocks’)

    How can I make the logger replace both of the above prints?
    I”m adding the logger to a program where many of the print() calls use end=
    From the logger documentation, I don’t see anything like end= for the logger

    Brian Okken
    @okken

    [Adam Parkin, Test Podcast] So I want to run coverage.py on my work project, but here’s a wrinkle: most of our tests are integration tests, so they spin up a set of Docker containers (like each REST API is in a separate running container), and then the tests make HTTP calls to the various services and assert on results.

    That’s going to mean that coverage won’t see what parts of the services get exercised by tests doesn’t it? Like if I have a test that looks like:

    `\def test_thing():
    result = requests.get('http://localhost:1234/some/path/on/a/container')

    assert something\_about\_the\_result```

    Since coverage.py\ is running in the container where that test is running and not where the web server is running, all it sees is that the test made a http call to some endpoint somewhere, right? Like there’s no way to tell coverage “oh wait, http://localhost:1234/some/path/on/a/container is actually part of our project, so figure out what code in another container is running and do coverage over that” or is there? Anyone have any experience or ideas on how to get coverage information in this situation?

    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] @Adam Parkin That got me thinking a bit about the purpose of coverage.. poking around I think this stackoverlow answer was actually pretty good: https://sqa.stackexchange.com/questions/24997/why-do-code-coverage-of-integration-test
    [David Kotschessa, Test Podcast] in particular
    • Unit tests are a much more effective way to exercise code than integration tests. Compared to integration tests, unit tests are lighter weight, faster, and more targeted. Using an integration test to cover code is like using a sledge hammer to drive in a nail.
    • Unit tests are about exercising code, whereas integration tests are about testing assumptions. When we write unit tests, we mock/fake/stub out the interfaces that the code depends on, and in the process we encode assumptions about how those interfaces behave. Integration tests are where we find out whether those assumptions were valid.
    [David Kotschessa, Test Podcast] "integration tests are about testing assumptions." hit home
    [Adam Parkin, Test Podcast] > Unit tests are about exercising code, whereas integration tests are about testing assumptions
    That’s… a really good way to think about it and articulates my feelings really well.
    [David Kotschessa, Test Podcast] yeah
    [Adam Parkin, Test Podcast] I dunno if that helps me answer my original question, but thanks for sharing that, super insightful.
    [David Kotschessa, Test Podcast] and yet I also get not wanting to go overboard and be hell bent on having a test for every.single.function
    [David Kotschessa, Test Podcast] sometimes a test that takes place inside the container you are hitting and runs a few of those functions - who knows what to call it - unit, component, integration? Might still serve a very good purpose
    [Brian Okken, Test Podcast] Wow. I don’t think I can disagree more.
    Brian Okken
    @okken
    [Adam Parkin, Test Podcast] That’s really interesting, I’d be interested in hearing your thoughts as to why.
    [Brian Okken, Test Podcast] Code coverage using unit tests is trivial. Just write little tests for all of your code. Done.
    [Brian Okken, Test Podcast] But what I really want code coverage to tell me is what parts of the system are actually being used.
    [Brian Okken, Test Podcast] So coverage while running tests that hit API points or even system tests seem more effective for that purpose.
    [Brian Okken, Test Podcast] Your question seemed more like “how do I run coverage on code sitting on a different machine”.
    [Brian Okken, Test Podcast] The answer of “run it on the same machine, if you can” is still reasonable. I think. I actually don’t know if you can split coverage onto a different machine. But if you can spin up a local server for testing purposes, on the same machine as the tests, the integration tests still work.
    [Brian Okken, Test Podcast] YMMV, of course, depending on your architecture.
    Brian Okken
    @okken

    [Adam Parkin, Test Podcast] Unfortunately in my specific case here that (running the server locally alongside the test) is not an option (or at least not without a lot of work to make that happen, which maybe that’s the answer :shrug).

    But yes, effectively it is “test is running on one machine, code under test is running on a different machine”.

    [Adam Parkin, Test Podcast] And so my dilemma is I want to gain insight into which parts of the overall system (which is made up of many services/APIs) are covered/exercised by the test suite and which are not (so as to find gaps that need addressing).
    Brian Okken
    @okken

    [Adam Parkin, Test Podcast] It occurs to me I’m probably misusing the term “integration test” here, in my case my tests are really “full system tests”, so maybe to tweak my question slightly: is it a good idea to generate coverage information for tests that are full system tests, and if so (or at the very least if it’s not a bad idea) how do you achieve that?

    Like maybe a more concrete way of thinking of this: say you have a bunch of selenium tests that you execute against your staging environment. You want to get a sense of what parts of the overall system are exercised by those tests and which are not, how would you go about discovering that?

    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] Yes @brian I know you are not a fan of the traditional "unit test" and the pyramid and all that, and I'm totallt sympathetic to your way of looking at it. That's why I followed up with my second statement, but I'm not sure where else to go from there in the context of the architecture in question.
    [Brian Okken, Test Podcast] I feel I need to preface my opinion with “I don’t do a lot of testing of web services”.
    [Brian Okken, Test Podcast] But I do test complex systems.
    Brian Okken
    @okken
    [Brian Okken, Test Podcast] At some level there is a thing that gets deployed that is larger than a single function but smaller than “the system”. These things have interfaces, and seem like they should be testable as components from their interface. I think that’s the sweet spot of testing complex systems. Test all the parts (services?) with feature testing. I really like the feature testing article on the twitter dev blog, https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html
    [Brian Okken, Test Podcast] I’m not opposed to unit tests. But they are for developer sanity checking, in my use of them. They don’t tell me if the system works.
    [Brian Okken, Test Podcast] Please correct me if I’m wrong in the assumption that feature testing each service could use coverage effectively, and adding to that system level workflow tests would sufficiently test a system. Even a complex web API level system.
    [Brian Okken, Test Podcast] I personally love unit tests. But I’m also a classicist, and I think lots of people teaching TDD and testing are mockists. It’s the mockist approach to unit testing I’m opposed to. I’m also opposed to having everything fully unit tested by a dev team, firing QA, and having just the customers test the system.
    Brian Okken
    @okken
    [Brian Okken, Test Podcast] The question of “can I run tests on one computer and collect coverage against code running on a different server” would be a great question for Ned Batchelder or even to the “testing in python” mailing list.
    Brian Okken
    @okken

    [David Kotschessa, Test Podcast] Say I want to do something crazy with tox like run...well... like every version of python since 2.7. (It's an experiment and possibly blog article).

    I'm confused bout the overlapping domains of:
    virtual environments (I use just venv)
    tox itself (which, I guess is lke venv, but you can still use it with venv?)
    Now I'm reading i might want pyenv if I want to install all these different versions of python.
    What's the simplest way?

    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] tox doesn’t install python versions. Also you have to specify libraries in the environments in tox.ini so you’ll not reuse the way you install them normally as a user from command line. tox is a lifesaver if you need a matrix of different dependency sets created and execute a (test) command in them. If you want to run the same thing for many different python versions it might not be the best tool.
    Brian Okken
    @okken
    [David Kotschessa, Test Podcast] If you want to run the same thing for many different python versions it might not be the best tool.
    Is that not considered one if it's primary uses?
    Brian Okken
    @okken
    [Tibor Arpas, Test Podcast] :) kind of. Maybe tell us more about your use case. Do you mean 12 versions of python 2.7 - 3.10? Are you sure a quick bash/python script wouldn’t be easiest?
    [Tibor Arpas, Test Podcast] tox docs says it uses virtualenv to create and manage many virtualenvs. I use tox with many different pythons versions and didn’t need pyenv in that context.
    [David Kotschessa, Test Podcast] It's not a real world use case. It's more of a giant experiment and learning experience involving writing code and watching it pass/fail in different versions to learn about the breaking changes and new features that came with each one.
    [David Kotschessa, Test Podcast] idea for an article
    [Tibor Arpas, Test Podcast] so, venv and tox overlap a little but I think easiest is to use tox (with it’s standard way to create virtualenvs and install dependencies)
    Brian Okken
    @okken

    [David Kotschessa, Test Podcast] So I guess here's where I'm confused. say I'm in an activated venv and I install a package, say django - it installs itself in the venv/bin/whatever folder, but not otherwise on my machine.
    The python version I'm using is also in venv/bin/python (or whatever) based on what version I'm using. Bu it's also installed globally.

    So I have python 2.7 (because mac still ships with it) and 3.8 (because that's what I installed)
    Soooo say I want to install python 3.2 - is there an installation method that puts it in the venv but does not install globally?

    [David Kotschessa, Test Podcast] because I'm only going to use it for that project.
    [Tibor Arpas, Test Podcast] tox creates all the configured virtual environments under .tox directory and I think by default doesn’t take anything from global/active virtualenv.
    [Tibor Arpas, Test Podcast] i think it’s fine to have tox globally… and then all the things which you want experiment with are configured in tox.ini and corresponding virtualenv are independent from global install, from pyenv or venv