Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
scholtalbers
@scholtalbers
thx for the patience - will have to continue this later
Marius van den Beek
@mvdbeek
sure, np
John Chilton
@jmchilton
@mvdbeek I have a worry that has been growing. Do you think auto-propagating the group:* tags is a mistake - because we are too aggressive with our tag propagation. The simplest way to think about it is a imagine a collection with two group tags in the HDAs, then you run a mapping step or something and resulting collection has those group tags on the HDCA itself. This so far is fine - the HDCA does have those groups so that sort of makes sense, then imagine running a structured_like tool on the result - the input to the tool is the HDCA - so every resulting HDA will get all the auto propagated tags (name tags and group tags). I think anyway. Can we do a better job auto propagating or do we just need to disable it or hide it behind a flag for 18.09? I was also contemplating group and autogroup tags - to the tool they could look the same but they'd behave different in this behavior
I'm ultimately nervous about committing to this behavior until we see it works for a broad range of workflows - maybe just sticking with your tagging tool for 18.09 is the way to go
Marius van den Beek
@mvdbeek
Going with the tagging tool / the apply rules tool sounds perfectly reasonable to me
for a first shot, that is
the concerns you have also apply to name tags, which already have this behavior, and for some scenarios like the one you described they just dont work
John Chilton
@jmchilton
is there a tweak to the propagation we can do to fix this all - like prevent collection contents from propagating up to HDCAs? Fixing nametags would probably fix group tags then also
Or do we just not have enough metadata to infer what should happen and we should give up
Marius van den Beek
@mvdbeek
It would always break down for scenarios where we don't know which tag should propagate. If you subtract input1 from input2 (we do that quite often with bedtools for instance) -- which tag should the output have ?
I think that depends on the context
So in workflows I drop certain tags from an output using PJAs, that is quite flexible and I haven't had the scenario where that didn't work
John Chilton
@jmchilton
Is that for getting nametags working properly or are you already using the newer stuff?
Marius van den Beek
@mvdbeek
Up to now only nametags, for the deseq2 workflow everything works absolutely beatiful
but the PJAs should work for all kinds of tags
btw a colleague of mine asked me recently why he can't name outputs in the tool form -- maybe tool form PJAs would be worth thinking about ?
John Chilton
@jmchilton
100% agree - anything to capture more metadata for valid workflow extraction from histories
galaxybot
@galaxybot
[Ferris] If one were to move a Galaxy install from one directory to the other, what would one have to do to get conda to move as well? Conda keeps trying to use python from the old directory
Marius van den Beek
@mvdbeek
setconda_prefix in the galaxy.ini / galaxy.yml to the new conda directory
Nate Coraor
@natefoo
does anyone know travis-fu to run something (a pre-stage?) that can determine what additional stages should be run and set something to do that conditionally?
maybe circleci is a better fit for this
Nicola Soranzo
@nsoranzo
Nate Coraor
@natefoo
yeah, i've looked in to the stages but i can't see a way to have one stage set something that can be used in a conditional (either via if: or in a script) in a later stage
Nicola Soranzo
@nsoranzo
I don't think that's possible in Travis
Nate Coraor
@natefoo
yeah. =( i can have it check on each executor (job? the terminology is muddled) but that requires a lot of repetition. for something you already "know"
although repetition is not exactly uncommon in CI. =)
Nicola Soranzo
@nsoranzo
Indeed
Marius van den Beek
@mvdbeek
what is it for ? is there not a chance a single build would be enough ? those can share artifacts with the deploy stage IIRC
Nicola Soranzo
@nsoranzo
And you can use macros in .travis.yml, if that helps
Nate Coraor
@natefoo
it's for starforge wheel building - wheels are either platform, pure python (needs seperate 2 and 3 builds), or pure python universal (2 and 3 in one wheel)
Nicola Soranzo
@nsoranzo
Look for py3_addons in https://github.com/galaxyproject/galaxy/blob/dev/.travis.yml for macro syntax.
Nate Coraor
@natefoo
right now over in galaxyproject/starforge-recipes it can build platform ones w/ matrix jobs out to linux and macos, 2.7, 3.4, 3.5, 3.6
@nsoranzo: oh using standard YAML references?
Nicola Soranzo
@nsoranzo
Speaking of which, we need to build wheels for the recently-released uwsgi 2.0.17.1 , building the previous 2.0.17 on Fedora 28 is broken due to its new glibc
@natefoo Yes (I think)
Nate Coraor
@natefoo
yeah it should be possible in its current state
Helena Rasche
@erasche
wrote a queue time SQL script? I'm pretty sure @natefoo sent me one at one point, but, here's a rewrite bceause I couldn't find that one:
CREATE TEMPORARY TABLE temp_queue_times AS
select
    min(a.create_time - b.create_time) as queue_time
from
    job_state_history as a
inner join
    job_state_history as b
on
    (a.job_id = b.job_id)
where
    a.job_id in (select id from job where tool_id like '%"$1"%' and state = 'ok' and create_time > (now() - '3 months'::interval))
    and a.state = 'running'
    and b.state = 'queued'
group by
    a.job_id
order by
    queue_time desc
;
--
select
    min(queue_time),
    percentile_cont(0.95) WITHIN GROUP (ORDER BY queue_time) as perc_95,
    percentile_cont(0.99) WITHIN GROUP (ORDER BY queue_time) as perc_99,
    max(queue_time)
from temp_queue_times;
("$1" because this was from a bash script and I forgot to replace that.BBut y'all get the idea.)
Looking at last 3 months of rna star jobs on our server:
 bash scripts/queue_time.sh rnastar
       min       |     perc_95     |     perc_99     |          max
-----------------+-----------------+-----------------+-----------------------
 00:00:08.007632 | 04:42:01.661542 | 10:59:26.373311 | 1 day 05:24:10.246863
ping @bgruening it's in galaxy's homedir.
Nate Coraor
@natefoo
yay!
not sure if i saved that anywhere either
Marius van den Beek
@mvdbeek
Cool, the build triggers are working. If you want to run additional python3 tests on a PR (api, framework,integration) comment with @galaxybot test py3
Jennifer Hillman-Jackson
@jennaj
usegalaxy.eu error reported at Biostars. I'm not familiar with the tool and don't have admin access to look at their error in detail. Could someone from EU team help? https://biostar.usegalaxy.org/p/29168/ Tx!
Marius van den Beek
@mvdbeek
I think https://bugs.python.org/issue21161 might be a problem for cheetah templates that use list comprehensions on python 3 :(
or at least for list comprehensions that reference variables from the surrounding scope, like [i for i in $var1 if i in $var2]
Vahid
@VJalili
@nsoranzo when I remove a pinned required from dependencies/pipfiles/default/pinned-requirements.txt and run make update-dependencies, the command then empties the dependencies/pipfiles/default/pinned-requirements.txt file. Is this an expected behavior?
Vahid
@VJalili
I think it is due to some conflicts in the dependencies.