Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jachym Cepicky
    @jachym
    any feedback or thoughs welcome
    Jachym Cepicky
    @jachym
    chm, the daemon still needs some polishing, just concept
    Jachym Cepicky
    @jachym
    just noting, that the daemon branch is now usable
    tested with 200 requests and set maxparallel to 20
    MacPingu
    @cehbrecht
    @jachym polling the database for stored jobs sounds more reliable then the current implementation. The pywps.processing module could be a common interface for the different job execution implementations. I think you already figured out how the scheduler part works: dump the job status and run the joblauncher with this status document (json) on a remote batch node. A shared file-system and the postgres DB are used to get the outputs and update the job status. The drmaa library is only the interface to schedulers like slurm. We might even skip it because it doesn’t look well maintained. We would then call slurm directly (skipping support for other scheduler systems like GridEngine).
    Jachym Cepicky
    @jachym

    thanks for feedback @macpingu, I'm finalising the pull request.

    also the dockersheduler should go there imho

    MacPingu
    @cehbrecht
    @jachym I got a bit confused. Did you tag v4.2.3 or was it accidently me? The version number was not adapted (geopython/pywps#500) and the changelog was not updated. There is also no upload to pypi. Should we update the tag?
    @huard @jachym I’m holding back the conda build for v4.2.3 until we have figured out what to do.
    conda-forge/pywps-feedstock#7
    Long Vu
    @tlvu
    Any update on this 4.2.3 release to pypi and conda? Thanks.
    MacPingu
    @cehbrecht
    @jachym should I update the 4.2.3 tag with a fix for version and changelog? I can’t upload to pypi … I have no permissions.
    Tom Kralidis
    @tomkralidis
    way to go all: https://twitter.com/luis_m_de_sousa/status/1191254829424947200
    MacPingu
    @cehbrecht
    @jachym @tomkralidis @ldesousa Can one of you give me a “go” on updating the 4.2.3 tag? Someone with permissions then need to do the upload on pypi.
    Tom Kralidis
    @tomkralidis
    +1/go
    MacPingu
    @cehbrecht
    @tomkralidis ok. Thanks :)
    MacPingu
    @cehbrecht
    I have updated the 4.2.3 tag … version number and changelog. Someone needs to upload the package to pypi. conda packages are builded.
    MacPingu
    @cehbrecht
    @jachym I have checked the pywps 4.2.3 on pypi with the Emu WPS. It works. Thanks :)
    https://pypi.org/project/pywps/4.2.3/
    Long Vu
    @tlvu
    Thanks for the 4.2.3 release @cehbrecht @jachym @tomkralidis
    MacPingu
    @cehbrecht

    @jachym thanks for the 4.2.4 release :)
    https://github.com/geopython/pywps/releases/tag/4.2.4

    Could you please upload it to pypi?

    The conda-forge package is build from github and a build for 4.2.4 is triggered:
    https://github.com/conda-forge/pywps-feedstock

    Trevor James Smith
    @Zeitsperre
    hi @cehbrecht, think we could push a new PyWPS in the coming days? Ouranosinc/raven#223 is reliant on the update mimetype fix.
    MacPingu
    @cehbrecht
    @Zeitsperre should it be a 4.2.5 release from a 4.2.x branch? The master is working different … we have not updated our deployments to use it yet.
    David Huard
    @huard
    @cehbrecht What's the status of the watchdog version ? Would you be comfortable using it in production ?
    MacPingu
    @cehbrecht
    @huard we first need to do some tests with it … probably in emu. The setup will change for the wps birds … and also the clean up of stalled jobs in pywps is still pending.
    David Huard
    @huard
    Ok, thanks. Is it safe to schedule the upgrade for the next release in ~ 3 months ?
    MacPingu
    @cehbrecht
    With some help from your side we will manage :)
    David Huard
    @huard
    Ok, I've asked my colleague Long to plan some time over the next months to work with you on this.
    MacPingu
    @cehbrecht
    Ok. Thanks :)
    Tom Kralidis
    @tomkralidis
    @cehbrecht what is your username on PyPI ?
    MacPingu
    @cehbrecht
    https://pypi.org/user/pingucarsti/
    Tom Kralidis
    @tomkralidis
    thanks, I’ve added you as a maintainer
    MacPingu
    @cehbrecht
    @tomkralidis thanks :)
    MacPingu
    @cehbrecht
    @tomkralidis @jachym @ldesousa I would like to make a patch release 4.2.5 from our release branch pywps-4.2. It has patches to get the scheduler extension working again. Ok?
    Tom Kralidis
    @tomkralidis
    ok
    MacPingu
    @cehbrecht
    @tomkralidis thanks :) Will do today.
    MacPingu
    @cehbrecht

    @jachym @ldesousa @tomkralidis I have released pywps 4.2.6:
    https://pypi.org/project/pywps/4.2.6/

    It has a patch for the scheduler extension and backports from the master branch (4.2.5). In 4.2.6 the travis tests are fixed and an import issue from a backport for the GPX validator.

    Luís de Sousa
    @ldesousa
    Very good @cehbrecht perhaps send an e-mail to the list to keep everyone updated. @jachym can you post an announcement on Twitter?
    MacPingu
    @cehbrecht
    @ldesousa e-mail done.
    Luís de Sousa
    @ldesousa
    I put up announcements on Twitter and Mastodon from my personal account:
    https://twitter.com/luis_m_de_sousa/status/1280037442477850624
    https://mastodon.social/@luis_de_sousa/104465678096528120
    @cehbrecht Thank you for keeping the ball rolling.
    Jorge Samuel Mendes de Jesus
    @jorgejesus
    The pywps twitter account seems a bit stalled
    who is taking care of it ?
    MacPingu
    @cehbrecht
    @jachym @huard Could we go one step back on the master? We have the python 3-only version and the new “watchdog” service on master. But the “watchdog” is still not finalised … and we are still using the 4.2.x version. It is a bit tricky to backport patches from master to 4.2.x … (has still python 2.x compatibility). I would prefer if we could move the current master to a 5.x-dev branch (with watchdog) and remove the watchdog from the master. We can release the master then as 4.4.0 (python 3 only). What do you think?
    David Huard
    @huard
    Ok for me.
    Long Vu
    @tlvu
    @cehbrecht if I remember correctly, the new "watchdog" service have to be activated in the config so could we simply leave it there on master because it is not active by default anyways? I just don't remember if the previous mechanism it replaces is still on master or not.
    Long Vu
    @tlvu
    But good call to go-live with one change at a time so it's easier to debug.
    MacPingu
    @cehbrecht
    @tlvu the implementation for the “watchdog, job queue” has changed … also introducing database migration. Currently we don’t use the master … still relying on 4.2.x … I would feel better if we can get the master in production again … and bring in the changes one by one: py3 only, database migration, job queue ...
    py3 only would be a 4.4.x release … database migration and jobqueue could go in 5.x.
    @tomkralidis @ldesousa do you have opinions on “moving jobqueue to dev branch”? How would you safely revert changes on master?