Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Nils Hempelmann
    @nilshempelmann
    @/all
    doodle and doc is set up for the next video conference, planed for May
    https://github.com/bird-house/bird-house.github.io/wiki/Meetings
    Nik Radoš
    @nikola-rados

    Hey all, I was hoping to get some help setting up birds in phoenix.

    I have deployed to a docker server using birdhouse-deploy and I've registered the available birds (hummingbird, finch, flyingpigeon, malleefowl) along with my own bird. The issue is that none of the bird's processes are working. When trying to run one I get an error message stating "Error: Could not read status document after 5 retries. Giving up.". This happens for all the birds and I'm not sure exactly how to fix it. I've looked through the code but don't have a strong understanding of what exactly the status document is. Any tips would be very much appreciated!

    Nils Hempelmann
    @nilshempelmann
    What kind of bird have you build ? Sounds good!
    Nik Radoš
    @nikola-rados
    @nilshempelmann I'm not familiar with birdy, I'll have to take a look. I am in the process of building a bird that wraps some internal tools from PCIC!
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados : birdy is a python client to call the services. easy to install, and handy to use https://github.com/bird-house/birdy
    @nikola-rados: sounds good for the PCIC bird. Curious to see code, if its allowed to share.
    Long Vu
    @tlvu
    @nikola-rados is your bird accessible by the anonymous Magpie user? curl --include "https://pavics.ouranos.ca/twitcher/ows/proxy/flyingpigeon?service=WPS&version=1.0.0&request=GetCapabilities"
    oh wait, how are you integrating your bird to the PAVICS stack or it is not integrated?
    Anyways, I have no experience with Phoenix, try giving birdy a try. It's another WPS client.
    The Jupyter env coming with the PAVICS stack has birdy already installed together with a bunch of other useful libraries. To try it quickly, you can try the one on our PAVICS instance: https://pavics.ouranos.ca/jupyter/
    Long Vu
    @tlvu
    I've emailed you the demo account password.
    Nik Radoš
    @nikola-rados

    Sorry I was afk for lunch.

    @nilshempelmann its in our github repo so feel free to have a look! It's still fairly early on so if you have any comments or suggestions those are welcome.

    @tlvu thanks for the info, I'll give it a try.

    Nils Hempelmann
    @nilshempelmann
    Cool. I ll check it out! Thanks.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados just installed thunderbird on my local mashin. Well as far as I see, it't empty so far.
    # import the WPS client and connet to the server
    from birdy import WPSClient
    import birdy 
    
    tb_server = 'http://localhost:8094/wps'   # link to the local thunderbird server
    
    # simple connection (not recommended for larger processing)
    tb = WPSClient(tb_server)
    
    # asyncron connection with progess status requests
    tb_i = WPSClient(url=tb_server, progress=True)
    Thats how you can call it with birdy. I put thunderbird on port 8094, per default it is on 5000.
    Nik Radoš
    @nikola-rados
    The only functionality is on a branch that has yet to be merged, i3-gen-climos. I'm not sure if this would change the outcome of this test.
    Nils Hempelmann
    @nilshempelmann
    it you want to test:
    out = tb.hello('Nikola')
    out.get()[0]
    good luck :-)
    Nik Radoš
    @nikola-rados
    Thank you, I'll give it a try when I get the chance!
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados I checked out the i3-gen-climos branch, but the environment.yml is not complete.
    ModuleNotFoundError: No module named 'nchelpers'
    Nils Hempelmann
    @nilshempelmann
    @huard @cehbrecht : What is the advantage in using notebooks for testing instead of the nosetest which are already integrated? For me it looks more like an advantage to test if a jupyter server is running. but this is PAVICS specific.
    Nik Radoš
    @nikola-rados
    @nilshempelmann I actually put the package into the requirements.txt file for pip rather than use conda.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados how do I than propperly install the bird?
    Nik Radoš
    @nikola-rados
    $ python3 -m venv venv
    $ source venv/bin/activate
    (venv)$ pip install -i https://pypi.pacificclimate.org/simple/ -r requirements.txt
    (venv)$ pip install .
    That sequence should do it! You can run thunderbird start from that environment.
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados I recieved an error:
    ERROR: Command errored out with exit status 1:
         command: /home/nils/anaconda3/envs/thunderbird/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rz37xr0m/netCDF4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rz37xr0m/netCDF4/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rz37xr0m/netCDF4/pip-egg-info
             cwd: /tmp/pip-install-rz37xr0m/netCDF4/
        Complete output (23 lines):
        Package hdf5 was not found in the pkg-config search path.
        Perhaps you should add the directory containing `hdf5.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'hdf5' found
        /tmp/pip-install-rz37xr0m/netCDF4/setup.py:161: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
          config = configparser.SafeConfigParser()
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-rz37xr0m/netCDF4/setup.py", line 341, in <module>
            _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
          File "/tmp/pip-install-rz37xr0m/netCDF4/setup.py", line 306, in _populate_hdf5_info
            raise ValueError('did not find HDF5 headers')
        ValueError: did not find HDF5 headers
        reading from setup.cfg...
        using nc-config ...
    
            HDF5_DIR environment variable not set, checking some standard locations ..
        checking /home/nils ...
        checking /usr/local ...
        checking /sw ...
        checking /opt ...
        checking /opt/local ...
        checking /usr ...
        ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    We should update our developer guide :-) https://birdhouse.readthedocs.io/en/latest/dev_guide.html#
    Nik Radoš
    @nikola-rados
    Ah yes, you need to install cdo as well, I forgot about that. We could move this conversation to PM's so we don't flood the chat.
    Nils Hempelmann
    @nilshempelmann

    Don't worry about the chat flood, its interesting for the others as well.
    the dependencies should be included. Better in the environment.yml to be able to install the thunderbird like the other birds with:
    conda env create
    make install
    thunderbird start

    After instelling cdo it looks like there is missing more: No package 'hdf5' found

    Nik Radoš
    @nikola-rados

    The reason I stuck with pip was because I wasn't sure how to apply the -i flag (pointing to the pcic pypi) to install our ce-dataprep package.

    I've seen that hdf5 error more times than I like haha. I think installing a few more things should fix it:

    sudo apt-get update && apt-get install -y \
        build-essential \
        cdo \
        # HDF5 libraries for cdo
        libhdf5-serial-dev \
        netcdf-bin \
        libnetcdf-dev

    That is the full process as far as I remember. Let me know if it works.

    Nik Radoš
    @nikola-rados

    @nilshempelmann using birdy with a local instance of thunderbird works fine. To test it I just run the hello process. I would also like to get an instance running on docker such that other people around the office would be able to test it out (plus I'd like to test heavier processing). I am able to successfully get capabilities using the url and port number but when I try and run a process using birdy this line loops endlessly:

    2020-03-24 12:47:44,973: hello [0/100] - PyWPS Process hello accepted

    So it seems the process "starts" but never actually executes (at least that is my assumption). Is there something I am not doing correctly here?

    David Huard
    @huard
    @nilshempelmann We're using both pytest unit tests and notebooks. Notebooks play the role of integration tests. So when developing, we run the unit tests, and when documenting, we run the notebooks.
    Nils Hempelmann
    @nilshempelmann
    @huard I don't fully understand. Integration test means to test the services in the PAVICS architecture?
    Nils Hempelmann
    @nilshempelmann
    @nikola-rados have a look here: https://github.com/bird-house/flyingpigeon/blob/master/environment.yml for the flags.
    David Huard
    @huard
    That and multiple services interacting together.
    Nils Hempelmann
    @nilshempelmann
    @huard If I get it right, the main difference for notebook-test are the option to take easily different input data. And a pool of test data will be established. Am I thinking in the right direction?
    David Huard
    @huard
    Not really ; ) pytest fixtures would be a better choice to check processes against a number of different input data. The notebook objective is make sure that typical "uses cases" work as expected and are documented. Compare the situation with a static rst tutorial, where changes to the code would make the tutorial non-functional until a user tries to follow the instructions and realizes it doesn't work.
    Nils Hempelmann
    @nilshempelmann
    @huard I see, so its more to test the documentation or better to keep it up to date. Than I understand why the notebook test are right placed in the docs folder.
    Thanks.
    David Huard
    @huard
    Genau !
    Long Vu
    @tlvu
    @nikola-rados In addition for notebooks to play the role of integration tests (all components working together, aka how a typical real user actually interact and experience the system), the notebooks also serve as tutorial notebooks, so free tutorial and testing the tutorials means they are ensured to always be up-to-date. See https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests#description for more infos.
    So basically view the unit tests as service for the developers while the notebooks as service for the real end users. They are complementary, not duplicate of each other.
    Nils Hempelmann
    @nilshempelmann
    @tlvu OK. I got it. well the unit test will focus on the function not on the processes anymore, that will be covered by the notebooks than
    Nik Radoš
    @nikola-rados
    @tlvu ok cool, thanks!
    Nils Hempelmann
    @nilshempelmann
    @tlvu I included a test-notebooks in the flyingpigeon make file
    If I got it right, that the client side birdy needs to be installed in the flyingpigoen env as well to run the notebook tests. Is there a way to keep server-side and client-side environments seperated?
    Long Vu
    @tlvu
    @nilshempelmann Finch have a requirements_dev.txtwhere additional dev related dependencies are specified: https://github.com/bird-house/finch/blob/4e9d2baaad0166a76c3e49481934c1eb56af007a/requirements_dev.txt#L7-L9
    Nils Hempelmann
    @nilshempelmann
    @tlvu D'accord.