galaxy-iuc/iuc

Chat with the Intergalactic Utility Commission of Galaxy tool developers

Robert Leach

@hepcat72

Let me look through it first. I hadn't thought to look there. Still new at this. Thanks for the tip.

John Chilton

@jmchilton

If there is an html file that might have a pretty version of the information as well

Robert Leach

@hepcat72

Yeah, I've been using the html file. In fact, I created a bash alias that automatically opens it when planemo test finishes.

But there's more in the json file...

Robert Leach

@hepcat72

Alright, I still can't find a problem. Have a look? https://gist.github.com/hepcat72/41ab5df0ee4f9a835e8111b634ee21de

Dan Fornika

@dfornika

@pvanheus great detective work tracking down that error. I'll take a closer look later today.

pvanheus

@pvanheus

Thanks @dfornika - as a test can you try and "flatten" the test-data - i.e. not have anything in a subdirectory of the test-data directory - and see if that makes Travis pass?

Philip Mabon

@Takadonet

Can anyone just take a peek at galaxyproject/tools-iuc#2324 ? Based on feedback we should be ready to go

pvanheus

@pvanheus

@Takadonet I added a couple of review comments but out of that it looks good.

Dan Fornika

@dfornika

@Takadonet @pvanheus I've also got a kraken2 data_manager in the works: https://github.com/dfornika/galaxytools/tree/master/data_managers/data_manager_build_kraken2_database I'll try to finish it up and submit it as soon as I can.

Philip Mabon

@Takadonet

@pvanheus thanks!

@dfornika Sounds good. We have a dozen users waiting to use it

pvanheus

@pvanheus

@dfornika yep we have a total hack at the moment that your work will replace. We have discovered that there are samples labelled M. tuberculosis in the public databases where ~ 5% of reads are identified by kraken2 as Mycobacteria. Not to say that the kraken2 standard database is perfect though (it can't reliably identify more granularity than genus Mycobacteria - this has been discussed on the relevant Slack).

Dan Fornika

@dfornika

It turned out to be a bit more work than I expected because I'd like to support (at least) four different modes of database creation: 'Standard: Build the standard db', 'Custom: choose which refseq categories to include, and add your own fasta files from galaxy', 'MiniKraken2: Download the MiniKraken2 database (v1 or v2)', 'Special: Build the special 16S databases RDP, GreenGenes, etc'.

pvanheus

@pvanheus

with "Custom" how do you provide the taxonomy information?

Dan Fornika

@dfornika

That's the one that I've spent the least time & testing on so far, so I'm not sure I have the right solution. This is what it does now, just downloads the NCBI taxonomy then adds the fasta then builds: https://github.com/dfornika/galaxytools/blob/bd602c325bc1b6c6453f89cf243bd9cb2dd0c520/data_managers/data_manager_build_kraken2_database/data_manager/kraken2_build_custom.py#L26

KyleL1998

@KyleL1998

@pvanheus I'm unable to see the review comments you have left on my kraken2 wrapper.

pvanheus

@pvanheus

@KyleL1998 can you see now?

KyleL1998

@KyleL1998

@pvanheus Yes thank you. I'll sort them out right now.

KyleL1998

@KyleL1998

@pvanheus After a git disaster I've pushed up the changes and they are going through the ci testing now

pvanheus

@pvanheus

and we are green @KyleL1998 - I don't have commit rights, but this has a +1 from me :)

pvanheus

@pvanheus

@dfornika did you get a chance to test my suggestion about the test-data?

Dan Fornika

@dfornika

@pvanheus not yet sorry I've had a busy day. Will try asap.

pvanheus

@pvanheus

no problems. its past 1am here so let me sleep

Jennifer Hillman-Jackson

@jennaj

Problems reported with installing mothur suite into a local galaxy (presumably 19.01). Any ideas about what is going on? Could someone jump in that could help? https://help.galaxyproject.org/t/problems-installing-mothur-suite/801

Dan Fornika

@dfornika

@pvanheus It looks like flattening the test-data directory did help but we're not quite passing the test suite. galaxyproject/tools-iuc#2252 Still a few differences between the expected output and the actual output, but at least the comparisons are being made cleanly now and it's possible to see where the output files differ.

pvanheus

@pvanheus

yeah I looked at that (around 4.30am :) ) - do the diff errors make sense to you? it still passes on my local machine... one option is to just bump the numbers of allowable lines difference..

my patch to Galaxy got accepted but it will only be out in 19.05

Aaron Petkau

@apetkau

@dfornika and @pvanheus I'm seeing a suspicious line in the Travis CI tests for snippy:

samtools sort: couldn't allocate memory for bam_mem

samtools is run with:

samtools sort -l 0 -T /tmp/snippy.5694. --threads 1 -m 8000M

Is it possible tests are failing on Travis because samtools is trying to request too much memory?

Aaron Petkau

@apetkau

There's a parameter --ram used by snippy that defaults to 8 GB. I'm wondering if this should be set in the tool using the GALAXY_MEMORY_MB environment variable (similar to setting cpus with GALAXY_SLOTS)?

Dan Fornika

@dfornika

Greate eye @apetkau . I'll give that a try.

lupetti2018

@lupetti2018

Thank you @jennaj . I'm looking for help to install Mothur on my local Galaxy. I tried to install the suite through the tool shed, but my Galaxy stops working if I attempt. Then I tried to install the single components through the repository, but it has troubles to install the mothur (1.39) dependency. In the terminal I’m getting this message:

PackageNotFoundError: Packages missing in current channels:

mothur
We have searched for the packages in the following channels:

I was running Galaxy 19.04, but I tried it now with the stable branch of the 19.01 Galaxy release with the same result. Both of them were a new install. I’ve tried to install other tools (with dependencies) and for them it worked.
I’m running Galaxy inside a virtualenv.

Dan Fornika

@dfornika

@lupetti2018 I think there may be some networking issues on the anaconda.org servers right now. I've also been having problems pulling packages from there, and I've seen a bit of discussion recently on the bioconda gitter channel saying that others are having problems.

Dave Clements

@tnabtaf

ACM BCB is being held sept 7-10 in Niagara Falls, NY. https://acm-bcb.org/2019/index.php
They have asked if we are interested in a tutorial and are willing to extend their deadline to accommodate a proposal.
Looking at last year's program, I don't think this would be a great match for a using tutorial. However it might be a great match for a tools workshop, as many talks are about new software.
Questions:

Agree/disagree?
Any interest in creating, submitting, and presenting a workshop?
This overlaps with GOBLET and INCOB, but that's all I know about.

Dan Fornika

@dfornika

@apetkau you were right! galaxyproject/tools-iuc#2252 I haven't quite got the GALAXY_MEMORY_MB variable working but setting --ram 1 allowed the tests to pass.

Dan Fornika

@dfornika

Ah, I think I see the issue. GALAXY_MEMORY_MB needs to be set via the resource manager. Is that configurable for the test environment?

Aaron Petkau

@apetkau

Awesome @dfornika. I've commented on pull request with a way to get it working. I don't think GALAXY_MEMORY_GB exists (only GALAXY_MEMORY_MB) but it can be converted to GB value.

pvanheus

@pvanheus

Thanks for the merge @dfornika - everyone, I think PR galaxyproject/tools-iuc#2252 is now ready for merging... but if anyone wants to give it another look, please do so!

Björn Grüning

@bgruening

@dfornika is https://github.com/galaxyproject/tools-iuc/pull/2252/files ready from your side?

lupetti2018

@lupetti2018

@dfornika thank you for your answer. Is there another way to install the packages on galaxy? I tried to install it via ephemeris, but I didn't have luck (I don't know if I have done it correctly)

pvanheus

@pvanheus

@lupetti2018 have you tried to re-install today? I think the networking issues are resolved.

lupetti2018

@lupetti2018

@pvanheus Yes I tried to reinstall today. Unfortunately it didn't work.

Nicola Soranzo

@nsoranzo

@lupetti2018 Can you paste here the conda create command executed by Galaxy, as reported in the logs?

lupetti2018

@lupetti2018

@nsoranzo this line? Executing command: /galaxy/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __mothur@_uv_ mothur
127.0.0.1 - - [18/Mar/2019:14:15:55 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 752|app: 0|req: 94/110] 127.0.0.1 () {46 vars in 996 bytes} [Mon Mar 18 14:15:55 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 14 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 0)
galaxy.workflow.run DEBUG 2019-03-18 14:15:55,595 [p:752,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Marking step 32 outputs of invocation 1 delayed (dependent collection [4] not yet populated with datasets)
galaxy.workflow.run DEBUG 2019-03-18 14:15:55,595 [p:752,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Marking step 33 outputs of invocation 1 delayed (dependent step [32] delayed, so this step must be delayed)
galaxy.workflow.scheduling_manager DEBUG 2019-03-18 14:15:57,946 [p:752,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Workflow invocation [1] scheduled

127.0.0.1 - - [18/Mar/2019:14:15:58 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 752|app: 0|req: 95/111] 127.0.0.1 () {46 vars in 996 bytes} [Mon Mar 18 14:15:58 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 11 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 2)
galaxy.workflow.scheduling_manager DEBUG 2019-03-18 14:15:58,962 [p:752,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Attempting to schedule workflow invocation [(1,)]

PackageNotFoundError: Packages missing in current channels:

mothur

We have searched for the packages in the following channels:

Nicola Soranzo

@nsoranzo

Yep, but there should be a similar one before where the mothur version is specified

lupetti2018

@lupetti2018

@nsoranzo This one? galaxy/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __mothur@1.39.5 mothur=1.39.5
galaxy.queue_worker INFO 2019-03-18 11:05:33,372 [p:609,w:1,m:0] [Thread-1] Instance 'main.web.1' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.tools.search DEBUG 2019-03-18 11:05:33,372 [p:609,w:1,m:0] [Thread-1] Starting to build toolbox index.
galaxy.tools.search DEBUG 2019-03-18 11:05:33,727 [p:609,w:1,m:0] [Thread-1] Toolbox index finished (355.027 ms)
127.0.0.1 - - [18/Mar/2019:11:05:35 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 22/39] 127.0.0.1 () {46 vars in 996 bytes} [Mon Mar 18 11:05:35 2019] POST /admin_toolshed/repository_installation_status_updates => generated 222 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 1)
127.0.0.1 - - [18/Mar/2019:11:05:38 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 23/40] 127.0.0.1 () {46 vars in 997 bytes} [Mon Mar 18 11:05:38 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 10 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 0)
127.0.0.1 - - [18/Mar/2019:11:05:41 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 24/41] 127.0.0.1 () {46 vars in 997 bytes} [Mon Mar 18 11:05:41 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 16 msecs (HTTP/1.1 200) 4 headers in 110 bytes (2 switches on core 3)
127.0.0.1 - - [18/Mar/2019:11:05:44 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 25/42] 127.0.0.1 () {46 vars in 997 bytes} [Mon Mar 18 11:05:44 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 11 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 1)
127.0.0.1 - - [18/Mar/2019:11:05:47 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 26/43] 127.0.0.1 () {46 vars in 997 bytes} [Mon Mar 18 11:05:47 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 10 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 0)
127.0.0.1 - - [18/Mar/2019:11:05:50 +0200] "POST /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "http://127.0.0.1:8080/admin_toolshed/prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
[pid: 609|app: 0|req: 27/44] 127.0.0.1 () {46 vars in 997 bytes} [Mon Mar 18 11:05:50 2019] POST /admin_toolshed/repository_installation_status_updates => generated 2 bytes in 9 msecs (HTTP/1.1 200) 4 headers in 110 bytes (1 switches on core 3)

PackageNotFoundError: Packages missing in current channels:

mothur 1.39.5*

Subina Mehta

@subinamehta

Could anyone suggest a good aligner for metagenomic datasets?

Nicola Soranzo

@nsoranzo

@lupetti2018 Yes, thanks. Let me have a look.

@lupetti2018 Also, what's the output of /galaxy/database/dependencies/_conda/bin/conda --version

Where communities thrive

People

Repo info

Activity