galaxy-iuc/iuc

Chat with the Intergalactic Utility Commission of Galaxy tool developers

Björn Grüning

@bgruening

Whenever we can we should avaoid producing RData and passing it around.

It can be a security nightmare and its not clear if you can reuse those object with other R versions imho, there is no guarentee

A while ago I look at http://parquet.apache.org/documentation/latest/

and I think in the long run we really should settle on something like that as an exchange format

M Bernt

@bernt-matthias

@bgruening good point. given a parquet data type there could even be a decent visualisation on the galaxy side. but I guess that this would bring 1/2 dependencies. at least pyarrow and maybe pandas

Björn Grüning

@bgruening

Yeah, but both are probably worth it

its just an idea that I play around with since a few month and probably a lot of work

so don't let me stop you

just wanted to through out that RData and pickles are not ideal

and we should have a plan to get rid of them

M Bernt

@bernt-matthias

I remember that feather came up a while ago here. Don't know if the format qualifies for long term storage. How about parquet?

Björn Grüning

@bgruening

All my research yielded me to parquet. But I can be wrong here of course.

Ajit Singh

@AjitPS

Thanks @mvdbeek , the last conda install logs when I tried GoSeq yesterday using the channel order iuc,conda-forge,bioconda,r,defaults show: conda create -y --override-channels --channel iuc --channel bioconda --channel r --channel defaults --channel conda-forge --name __bioconductor-org.hs.eg.db@3.7.0 bioconductor-org.hs.eg.db=3.7.0
PackageNotFoundError: Package not found: Conda could not find '
Removing failed conda install of bioconductor-org.hs.eg.db, version '3.7.0'
Same happens then for bioconductor-org.dm.eg.db 3.7.0, bioconductor-org.dr.eg.db 3.7.0, bioconductor-org.mm.eg.db 3.7.0 and r-dplyr 3.7.0

Later conda create -y --override-channels --channel iuc --channel bioconda --channel r --channel defaults --channel conda-forge --name __r-ggplot2@3.1.0 r-ggplot2=3.1.0 worked fine, as did the r-optparse=1.6.0 install.
once that's done, it shows: Using dependency bioconductor-goseq version None of type conda
conda create -y --override-channels --channel iuc --channel bioconda --channel r --channel defaults --channel conda-forge --name __bioconductor-org.hs.eg.db@_uv_ bioconductor-org.hs.eg.db
Package plan for installation in environment /home/usern/galaxy/database/dependencies/_conda/envs/__bioconductor-org.hs.eg.db@_uv_ and so on

Nicola Soranzo

@nsoranzo

If you are on conda 4.6.x, there's a bug which prevents installation of bioconductor-org.hs.eg.db, see conda/conda#8306

Ajit Singh

@AjitPS

thanks, will have a look

M Bernt

@bernt-matthias

Irrespective of the backend (parquet, sql, tsv, json, ... ) I see the biggest problem in a (tool developer)friendly way to serialize complex data types. Just imagine a list consisting of data frames, matrices, named inter vectors, unnamed double vectors. Seems a nightmare.

Ajit Singh

@AjitPS

@nsoranzo thanks, the conda installed on our Galaxy is conda 4.3.15, can update to newer version if this was fix in later ones (e.g, conda 4.6.7)

Nicola Soranzo

@nsoranzo

@AjitPS Actually I thought the downgrading from 4.6 to 4.5 would help, but that's not the case unfortunately.

Ajit Singh

@AjitPS

thanks @nsoranzo

Jochen Bick

@bimbam23

how to do you add a display_application to galaxy? https://github.com/bimbam23/AnnOverlappeR/tree/master/Galaxy_app/display_applications/moadb

Jochen Bick

@bimbam23

-to

M Bernt

@bernt-matthias

Is it possible to <filter/> output based on the type of an input dataset? I have a program that

outputs type A for input of type A
outputs type B for input of type C or D
so its not a 1:1 mapping :(. I would like to have the inputs in a single param that allows A and B as inputs.

Alternatively, could this be done with the <change_format/>?

Björn Grüning

@bgruening

Is A, C and D just a filetype or a select field?

M Bernt

@bernt-matthias

its the file type of the only input field I have for this tool

Jennifer Hillman-Jackson

@jennaj

Hi all - Are there plans to update the GATK wrappers or has that project delayed/abandoned (for now)? Ref galaxyproject/tools-iuc#194

Björn Grüning

@bgruening

@jennaj: @davebx was working on this afaik

Robert Leach

@hepcat72

I have 3 output collections (one is list, one is list:paired, and one is list:list). I am using a filter and discover_datasets in their creation. Only one is ever populated. I only want the populated one to be added to the user's history, but currently, they are all added even though one is populated and the others are empty. Is there a way to prevent the empty ones from being added. BTW, the populated one corresponds to the user's selected run type (paired-end, single-end, or "complex".

Robert Leach

@hepcat72

Here's the xml...

<outputs>
    <data format="tabular" name="summary" label="${tool.name} on ${on_string}: Summary" />
    <collection name="split_output_complex" type="list:list" format_source="input" label="${tool.name} on complex ${on_string}">
        <filter>run_type == 'complex'</filter>
        <discover_datasets pattern="(?P&lt;identifier_0&gt;.*?)-XcomplexX-(?P&lt;identifier_1&gt;\S+)\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
    <collection name="split_output_single" type="list" format_source="input" label="${tool.name} on single-end ${on_string}">
        <filter>run_type == 'single'</filter>
        <discover_datasets pattern="(?P&lt;identifier_0&gt;.*?)-XsingleX-\S+\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
    <collection name="split_output_paired" type="list:paired" format_source="input" label="${tool.name} on paired-end ${on_string}">
        <filter>run_type == 'paired'</filter>
        <discover_datasets pattern="(?P&lt;identifier_1&gt;.*?)-XpairedX-(?P&lt;identifier_0&gt;\S+)\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
</outputs>

Marius van den Beek

@mvdbeek

Not sure how filters and collections work together, but if this were normal datasets I'd guess the filter isn't working

maybe try casting the parameter to a string first ?

<outputs>
    <data format="tabular" name="summary" label="${tool.name} on ${on_string}: Summary" />
    <collection name="split_output_complex" type="list:list" format_source="input" label="${tool.name} on complex ${on_string}">
        <filter>str(run_type) == 'complex'</filter>
        <discover_datasets pattern="(?P&lt;identifier_0&gt;.*?)-XcomplexX-(?P&lt;identifier_1&gt;\S+)\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
    <collection name="split_output_single" type="list" format_source="input" label="${tool.name} on single-end ${on_string}">
        <filter>str(run_type) == 'single'</filter>
        <discover_datasets pattern="(?P&lt;identifier_0&gt;.*?)-XsingleX-\S+\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
    <collection name="split_output_paired" type="list:paired" format_source="input" label="${tool.name} on paired-end ${on_string}">
        <filter>str(run_type) == 'paired'</filter>
        <discover_datasets pattern="(?P&lt;identifier_1&gt;.*?)-XpairedX-(?P&lt;identifier_0&gt;\S+)\.(?P&lt;ext&gt;.*)" directory="split" visible="false" />
    </collection>
</outputs>

Robert Leach

@hepcat72

Thanks, I'll know shortly if that did it

Huh. They still all show up in the history... hmmmm

This run was using the 'complex' value:

Item 5 is populated, but 6 & 7 are empty

Robert Leach

@hepcat72

Maybe I'm not referencing the run_type value correctly...

Marius van den Beek

@mvdbeek

In which case it'd be surprising that you get the outputs, you'd expect either an error or no outputs

but you could check that the filtering on discovered collections works at all by just writing <filter>False</filter, that should remove that output

if that doesn't work it's worth opening an issue on the Galaxy repo

Robert Leach

@hepcat72

I'm trying the False test. Will report

OK. False prevents the collection from being added. Has to be something wrong with my code.

Marius van den Beek

@mvdbeek

Cool, yeah, then maybe you're not referencing the run_type parameter correctly

is that in a conditional or a section ?

if yes you'd reference the value with conditional_name['run_type']

Robert Leach

@hepcat72

<conditional name="how_to_name_read_types">
    <param label="Run Type" name="run_type" type="select">
        <option selected="true" value="single">Single-End</option>
        <option value="paired">Paired-End</option>
        <option value="complex">Complex</option>
    </param>
    <when value="complex">
...

Marius van den Beek

@mvdbeek

yup, try how_to_name_read_types['run_type]

Robert Leach

@hepcat72

OK. Thanks

OK, that worked. Thanks!

Python's not my native tongue. ;)

Where communities thrive

People

Repo info

Activity