Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Nils Hempelmann
    @nilshempelmann
    good luck :-)
    Nik Radoš
    @nikola-rados
    Thank you, I'll give it a try when I get the chance!
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados I checked out the i3-gen-climos branch, but the environment.yml is not complete.
    ModuleNotFoundError: No module named 'nchelpers'
    Nils Hempelmann
    @nilshempelmann
    @huard @cehbrecht : What is the advantage in using notebooks for testing instead of the nosetest which are already integrated? For me it looks more like an advantage to test if a jupyter server is running. but this is PAVICS specific.
    Nik Radoš
    @nikola-rados
    @nilshempelmann I actually put the package into the requirements.txt file for pip rather than use conda.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados how do I than propperly install the bird?
    Nik Radoš
    @nikola-rados
    $ python3 -m venv venv
    $ source venv/bin/activate
    (venv)$ pip install -i https://pypi.pacificclimate.org/simple/ -r requirements.txt
    (venv)$ pip install .
    That sequence should do it! You can run thunderbird start from that environment.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados I recieved an error:
    ERROR: Command errored out with exit status 1:
         command: /home/nils/anaconda3/envs/thunderbird/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rz37xr0m/netCDF4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rz37xr0m/netCDF4/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rz37xr0m/netCDF4/pip-egg-info
             cwd: /tmp/pip-install-rz37xr0m/netCDF4/
        Complete output (23 lines):
        Package hdf5 was not found in the pkg-config search path.
        Perhaps you should add the directory containing `hdf5.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'hdf5' found
        /tmp/pip-install-rz37xr0m/netCDF4/setup.py:161: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
          config = configparser.SafeConfigParser()
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-rz37xr0m/netCDF4/setup.py", line 341, in <module>
            _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
          File "/tmp/pip-install-rz37xr0m/netCDF4/setup.py", line 306, in _populate_hdf5_info
            raise ValueError('did not find HDF5 headers')
        ValueError: did not find HDF5 headers
        reading from setup.cfg...
        using nc-config ...
    
            HDF5_DIR environment variable not set, checking some standard locations ..
        checking /home/nils ...
        checking /usr/local ...
        checking /sw ...
        checking /opt ...
        checking /opt/local ...
        checking /usr ...
        ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    We should update our developer guide :-) https://birdhouse.readthedocs.io/en/latest/dev_guide.html#
    Nik Radoš
    @nikola-rados
    Ah yes, you need to install cdo as well, I forgot about that. We could move this conversation to PM's so we don't flood the chat.
    Nils Hempelmann
    @nilshempelmann

    Don't worry about the chat flood, its interesting for the others as well.
    the dependencies should be included. Better in the environment.yml to be able to install the thunderbird like the other birds with:
    conda env create
    make install
    thunderbird start

    After instelling cdo it looks like there is missing more: No package 'hdf5' found

    Nik Radoš
    @nikola-rados

    The reason I stuck with pip was because I wasn't sure how to apply the -i flag (pointing to the pcic pypi) to install our ce-dataprep package.

    I've seen that hdf5 error more times than I like haha. I think installing a few more things should fix it:

    sudo apt-get update && apt-get install -y \
        build-essential \
        cdo \
        # HDF5 libraries for cdo
        libhdf5-serial-dev \
        netcdf-bin \
        libnetcdf-dev

    That is the full process as far as I remember. Let me know if it works.

    Nik Radoš
    @nikola-rados

    @nilshempelmann using birdy with a local instance of thunderbird works fine. To test it I just run the hello process. I would also like to get an instance running on docker such that other people around the office would be able to test it out (plus I'd like to test heavier processing). I am able to successfully get capabilities using the url and port number but when I try and run a process using birdy this line loops endlessly:

    2020-03-24 12:47:44,973: hello [0/100] - PyWPS Process hello accepted

    So it seems the process "starts" but never actually executes (at least that is my assumption). Is there something I am not doing correctly here?

    David Huard
    @huard
    @nilshempelmann We're using both pytest unit tests and notebooks. Notebooks play the role of integration tests. So when developing, we run the unit tests, and when documenting, we run the notebooks.
    Nils Hempelmann
    @nilshempelmann
    @huard I don't fully understand. Integration test means to test the services in the PAVICS architecture?
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados have a look here: https://github.com/bird-house/flyingpigeon/blob/master/environment.yml for the flags.
    David Huard
    @huard
    That and multiple services interacting together.
    Nils Hempelmann
    @nilshempelmann
    @huard If I get it right, the main difference for notebook-test are the option to take easily different input data. And a pool of test data will be established. Am I thinking in the right direction?
    David Huard
    @huard
    Not really ; ) pytest fixtures would be a better choice to check processes against a number of different input data. The notebook objective is make sure that typical "uses cases" work as expected and are documented. Compare the situation with a static rst tutorial, where changes to the code would make the tutorial non-functional until a user tries to follow the instructions and realizes it doesn't work.
    Nils Hempelmann
    @nilshempelmann
    @huard I see, so its more to test the documentation or better to keep it up to date. Than I understand why the notebook test are right placed in the docs folder.
    Thanks.
    David Huard
    @huard
    Genau !
    Long Vu
    @tlvu
    @nikola-rados In addition for notebooks to play the role of integration tests (all components working together, aka how a typical real user actually interact and experience the system), the notebooks also serve as tutorial notebooks, so free tutorial and testing the tutorials means they are ensured to always be up-to-date. See https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests#description for more infos.
    So basically view the unit tests as service for the developers while the notebooks as service for the real end users. They are complementary, not duplicate of each other.
    Nils Hempelmann
    @nilshempelmann
    @tlvu OK. I got it. well the unit test will focus on the function not on the processes anymore, that will be covered by the notebooks than
    Nik Radoš
    @nikola-rados
    @tlvu ok cool, thanks!
    Nils Hempelmann
    @nilshempelmann
    @tlvu I included a test-notebooks in the flyingpigeon make file
    If I got it right, that the client side birdy needs to be installed in the flyingpigoen env as well to run the notebook tests. Is there a way to keep server-side and client-side environments seperated?
    Long Vu
    @tlvu
    @nilshempelmann Finch have a requirements_dev.txtwhere additional dev related dependencies are specified: https://github.com/bird-house/finch/blob/4e9d2baaad0166a76c3e49481934c1eb56af007a/requirements_dev.txt#L7-L9
    Nils Hempelmann
    @nilshempelmann
    @tlvu D'accord.
    Nils Hempelmann
    @nilshempelmann
    @tlvu I made it with the notebook tests... almost :-)
    Do you have a trick to avoid this error:
    assert reference_output == test_output failed:
    
      '<img src="ht...width="400"/>' == '<img src="ht...width="400"/>'
      - <img src="http://127.0.0.1:8093/outputs/STATUS_FILE/tmpirlo_k7d.png" width="400"/>
      ?                                                        ^^  ----
      + <img src="http://127.0.0.1:8093/outputs/STATUS_FILE/tmplwz7aplo.png" width="400"/>
      ?                                                        ^^^^^^
    different output filenames on purpose. To keep the notebook also as the original idea for documentation and examples, the files should be shown.
    @tlvu @cehbrecht We should integrate the notebook-test also in the main documentaion:
    https://birdhouse.readthedocs.io/en/latest/dev_guide.html#writing-tests
    Long Vu
    @tlvu

    @nilshempelmann

    @tlvu I made it with the notebook tests... almost :-)
    Do you have a trick to avoid this error:

    assert reference_output == test_output failed:
    
      '<img src="ht...width="400"/>' == '<img src="ht...width="400"/>'
      - <img src="http://127.0.0.1:8093/outputs/STATUS_FILE/tmpirlo_k7d.png" width="400"/>
      ?                                                        ^^  ----
      + <img src="http://127.0.0.1:8093/outputs/STATUS_FILE/tmplwz7aplo.png" width="400"/>
      ?                                                        ^^^^^^

    I'll push an regex update soon.

    Long Vu
    @tlvu
    @nilshempelmann new regex pushed.

    @tlvu @cehbrecht We should integrate the notebook-test also in the main documentaion:
    https://birdhouse.readthedocs.io/en/latest/dev_guide.html#writing-tests

    LOL that Writing Tests section is empty for now. Happy to chime in about testing notebooks if someone start that section.

    Nils Hempelmann
    @nilshempelmann
    @tlvu @huard I understand more and more how you did the notebooks integration tests.
    will start writing the doc section. Good moment to fill this gap :
    bird-house/birdhouse-docs#46
    Nils Hempelmann
    @nilshempelmann
    @tlvu Are you familliar with Failed to open UDUNITS-2 XML unit database. even if udunits2 is available in the environment.
    Long Vu
    @tlvu
    @nilshempelmann no, I think you added that udunits2 library so you probably have more knowledge than me.
    MacPingu
    @cehbrecht
    @nikola-rados Is you thunderbird working now? I just checked the master branch and tested “hello” with birdy … works. But … you need to use Python 3.7 or Python 3.6. PyWPS is not working with Python 3.8:
    geopython/pywps#508
    I would use the conda environment for heavy dependencies like netcdf etc.
    I have pinned Python 3.7 in the conda environment, like in Emu:
    https://github.com/bird-house/emu/blob/master/environment.yml
    MacPingu
    @cehbrecht
    @tlvu do you know the GeoHealthCheck monitoring service?
    https://geohealthcheck.org/
    Long Vu
    @tlvu
    @cehbrecht wow that GeoHealthCheck presentation looks great !
    Long Vu
    @tlvu
    LOL I am the first WPS demo on the demo site https://demo.geohealthcheck.org/resource/297?lang=en, it's Finch on my test server.
    My Finch failed the monitoring because of this existing bug bird-house/birdhouse-deploy#14
    Nik Radoš
    @nikola-rados
    @cehbrecht master branch doesn't really have anything on it atm. Its just the cookiecutter with the name thunderbird.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados if you want, join the next VC: https://github.com/bird-house/bird-house.github.io/wiki/Meetings#next-vc
    it is interesting to know what you are doing and easier to support in terms of questions