Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Marius
    @mvdbeek:matrix.org
    [m]
    yeah, the isinstance check doesn't work, it's wrapped in a streaming response
    not sure how something reasonable could look like ... subclass FileResponse ? Set the headers in each route ?
    Marius
    @mvdbeek:matrix.org
    [m]
    Backend meeting in 30 minutes!
    backend group meeting
    Tuesday, 15 November ยท 16:00 โ€“ 17:00
    Google Meet joining info
    Video call link: https://meet.google.com/myu-ianj-bke
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Hi - a bit of a cross-wg request here that we've discussed in the systems WG (I'll also post on ui-ux): one of our biggest nuisances with figuring out job failures is due to a couple of key bits of info missing from the dataset info page (and/or error report email). This includes stuff like the dependency or container resolution as of the job runtime (not when viewed) and the job command line (not the tool command line): galaxyproject/galaxy#13037
    Is there any chance this could get prioritized for 23.1?
    1 reply
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Oh, we agreed on that? Nice. ๐Ÿ˜
    And yes, for that.
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    It's not merged yet, but I think we did agree.
    dannon
    @dannon:matrix.org
    [m]
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    It's approved, we can merge it :)
    Marius
    @mvdbeek:matrix.org
    [m]
    Should we start building wheels for 3.11 for those projects that haven't built them yet ? I'm intrigued by the claimed speedups, and we should ideally support 3.11 for the next release
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    I was looking into it for a few days already: https://github.com/nsoranzo/galaxy/tree/python3.11
    Marius
    @mvdbeek:matrix.org
    [m]
    huh, the ci managed to build h5py, that's surpring
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    The main issue is that there isn't a numpy version that supports all versions from 3.7 to 3.11
    Same for scipy
    Marius
    @mvdbeek:matrix.org
    [m]
    when's 3.7 EOL ?
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    2023-06
    Marius
    @mvdbeek:matrix.org
    [m]
    hmm, can we pin a range of versions ?
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    In theory you could specify different numpy versions depending on the Python version for poetry, but this seems broken: python-poetry/poetry#5506
    I can open a draft PR so interested people can find the ongoing status, but I think it's fine to wait a couple of months so the binary dependencies are built upstream.
    Marius
    @mvdbeek:matrix.org
    [m]

    I've got it all built locally, issues were pysam (there's an open PR fixing the build on OSX), h5py (need to install dep via brew and set env var) and the numpy pin, which I've moved to

    -numpy==1.21.6 ; python_version >= "3.7" and python_version < "3.12"
    +numpy>=1.21.6,<1.23.5 ; python_version >= "3.7" and python_version < "3.12

    I guess we'll have to at least wait for a new pysam release, but that seems actively developed these days

    or not, since it's only a problem on OSX ... depends if the version range pin is acceptable
    Marius
    @mvdbeek:matrix.org
    [m]
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    Note that pysam 0.20.0 currently breaks a lot of our tests (had to revert it in galaxyproject/galaxy#14888 )
    May be easy to fix, didn't have time to dig.
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    Re. numpy, I'd use 2 lines:
    numpy<1.22; python <"3.8"
    numpy>=1.23.2; python >="3.8"
    John Davis
    @ic4f:matrix.org
    [m]
    in an effort to run mypy on a loaded galaxy venv (see galaxyproject/galaxy#15004), I'm trying to install both pinnned-requirements and pinned-lint-requirements. However, there's a conflict: lint requirements list importlib-metadata==4.2.0, whereas the other file lists importlib-metadata==4.13.0. Can we use the same pin for lint requirements (4.13.0), or is there a reason why they are different?
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    These pins are generated separately, and flake8 has a more restrictive pin: https://github.com/PyCQA/flake8/blob/main/setup.cfg#L44
    John Davis
    @ic4f:matrix.org
    [m]
    I suppose we could separate the two flake8 dependencies and mypy + types* (listed in lint-requirements) - would that make sense?
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    That should work, if you update tox.ini accordingly.
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    Not sure when this broke, but I noticed with one of my Earlham tools that when testing with planemo test --biocontainers the job command generated contains docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -e "GALAXY_MEMORY_MB=$GALAXY_MEMORY_MB" -e "GALAXY_MEMORY_MB_PER_SLOT=$GALAXY_MEMORY_MB_PER_SLOT" -e "HOME=$HOME" -e "_GALAXY_JOB_HOME_DIR=$_GALAXY_JOB_HOME_DIR" -e "_GALAXY_JOB_TMP_DIR=$_GALAXY_JOB_TMP_DIR" -e "TMPDIR=$TMPDIR" -e "TMP=$TMP" -e "TEMP=$TEMP" followed by the volume mounts, but the $TMPDIR directory (which is something like /tmp/tmpd2u36yzi/tmp under planemo test) is not mounted on the container, so when the tool runs mktemp -dt "foo.XXXXXXXXXX" it fails with mktemp: (null): No such file or directory
    Suggestions on the best way to fix this?
    Marius
    @mvdbeek:matrix.org
    [m]
    what do the volume mounts look like ?
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    After the 2 mounts for Galaxy root and the tool XML, there is: -v /tmp/tmp4vz7lyq3/job_working_directory/000/7:/tmp/tmp4vz7lyq3/job_working_directory/000/7:ro -v /tmp/tmp4vz7lyq3/job_working_directory/000/7/outputs:/tmp/tmp4vz7lyq3/job_working_directory/000/7/outputs:rw -v "$_GALAXY_JOB_TMP_DIR:$_GALAXY_JOB_TMP_DIR:rw" -v "$_GALAXY_JOB_HOME_DIR:$_GALAXY_JOB_HOME_DIR:rw" -v /tmp/tmp4vz7lyq3/job_working_directory/000/7/working:/tmp/tmp4vz7lyq3/job_working_directory/000/7/working:rw -v /tmp/tmp4vz7lyq3/files:/tmp/tmp4vz7lyq3/files:ro -v /usr/users/ga002/soranzon/software/galaxyproject_galaxy_release_22.05/tool-data:/usr/users/ga002/soranzon/software/galaxyproject_galaxy_release_22.05/tool-data:ro -v /usr/users/ga002/soranzon/software/galaxyproject_galaxy_release_22.05/tool-data:/usr/users/ga002/soranzon/software/galaxyproject_galaxy_release_22.05/tool-data:ro --cpus ${GALAXY_SLOTS:-1} -w /tmp/tmp4vz7lyq3/job_working_directory/000/7/working
    While TMPDIR is /tmp/tmp4vz7lyq3/tmp
    Marius
    @mvdbeek:matrix.org
    [m]
    Hmm, -v "$_GALAXY_JOB_TMP_DIR:$_GALAXY_JOB_TMP_DIR:rw" ... that should be enough, I thought the job script sets TMPDIR to _GALAXY_JOB_TMP_DIR ?!
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    I don't think that's the case
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    Apparently I am not the only one that found the TMPDIR issue: galaxyproject/tools-iuc#4931
    M Bernt
    @bernt-matthias:matrix.org
    [m]
    Marius
    @mvdbeek:matrix.org
    [m]
    Yeah, also broke a whole bunch of regular tests ... guess we got to mount all the env variables in. I don't love it for the lack of isolation, but I think as an admin you can set per job tmp dirs
    Marius
    @mvdbeek:matrix.org
    [m]

    No meeting today, thanksgiving week. I do have a bunch of items I'd appreciate your input / assignment on if you do have the time:

    galaxyproject/galaxy#15045
    galaxyproject/galaxy#15007
    galaxyproject/galaxy#15005
    galaxyproject/galaxy#14994
    galaxyproject/galaxy#13037

    and a review for galaxyproject/galaxy#15039 would be cool!
    1 reply
    John Davis
    @ic4f:matrix.org
    [m]
    Not urgent, but when someone has a minute, this needs a review galaxyproject/galaxy#14993
    Nicola Soranzo: I'll try to figure this out today galaxyproject/galaxy#15037
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    John Davis: That's the one after on my list, if you prefer to work on something else.
    1 reply
    John Davis
    @ic4f:matrix.org
    [m]
    sounds good. But let me know if anything changes and/or if I can help.
    Marius
    @mvdbeek:matrix.org
    [m]
    There;s also galaxyproject/planemo#1327 and galaxyproject/planemo#1329 that could use reviews
    Marius
    @mvdbeek:matrix.org
    [m]
    Quick reminder, backend meeting in 30 minutes:
    backend group meeting
    Tuesday, 29 November ยท 16:00 โ€“ 17:00
    Google Meet joining info
    Video call link: https://meet.google.com/myu-ianj-bke
    martenson
    @martenson:matrix.org
    [m]
    Power outage
    Brb