Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 26 20:54
    ericenns closed #1288
  • May 26 20:45
    deepsidhu85 synchronize #1261
  • May 26 20:33
    deepsidhu85 synchronize #1261
  • May 26 18:29
    deepsidhu85 synchronize #1261
  • May 26 17:37
    ericenns synchronize #1288
  • May 26 17:26
    deepsidhu85 synchronize #1261
  • May 26 17:24
    ericenns review_requested #1288
  • May 26 17:23
    ericenns closed #1286
  • May 26 17:22
    ericenns opened #1288
  • May 26 17:09
    joshsadam synchronize #1277
  • May 26 15:55
    apetkau labeled #1287
  • May 26 15:55
    apetkau labeled #1287
  • May 26 15:55
    apetkau opened #1287
  • May 26 15:55
    apetkau labeled #1287
  • May 26 15:44
    joshsadam synchronize #1286
  • May 26 14:25
    joshsadam review_requested #1286
  • May 26 14:25
    joshsadam assigned #1286
  • May 26 14:25
    joshsadam opened #1286
  • May 26 14:15
    ksierks synchronize #1285
  • May 26 13:23
    ericenns synchronize #1280
pvanheus
@pvanheus
disk space management is what I am mostly interested in
1 reply
btw huge success with Workbench i.e. our Docker-ized IRIDA today: we are teaching a week-long workshop on SARS-CoV-2 bioinformatics to people from 9 different African public health labs (well 10, including our remoting-in people today). one of the people showed up with 48 samples of SARS-CoV-2 they had just sequenced - they loaded them all up into IRIDA, hit go on the analysis and had results in half an hour. this is on a VM with 4 virtual CPUs.
pvanheus
@pvanheus
it did reveal something I'm not quite sure how to deal with though - the negative control failed its pipeline run because it (after) filtering) has literally zero reads - which causes QualiMap BamQC to fall over... which causes the pipeline as a whole to fail. I'm not quite sure how to deal with this - its an expected failure mode of BamQC (https://bitbucket.org/kokonech/qualimap/src/8bdb6b7414319ac595df198703460f70e3aa840f/src/main/java/org/bioinfo/ngs/qc/qualimap/process/BamStatsAnalysis.java#lines-533)... one hack would be to add in a step that added at least 1 "synthetic read" after filtering... any other ideas?
Aaron Petkau
@apetkau
That's awesome on the SARS-CoV-2 workshop. I'm glad it worked out
Which pipeline were you using by the way?
Yeah, pipelines failing because some tools fail due to conditions like too few reads is an issue in IRIDA/Galaxy
(Other examples is trying to process too few reads in shovill or spades)
Ideally it would be displayed nicer in IRIDA instead of just showing the same as a regular tool error.
Aaron Petkau
@apetkau
For your case, a single "synthetic read" could work. Or a wrapper around the tool which detects if the tool failed due to 0 reads and ignores the error if so (or does different error handling). But I don't think that's ideal
pvanheus
@pvanheus
these are our pipelines - Illumina - https://github.com/combat-sars-cov-2/irida-plugin-sars-cov-2-illumina and Nanopore - https://github.com/combat-sars-cov-2/irida-plugin-sars-cov-2-nanopore
apologies for the lack of README, will fix soon. the pipelines JARs are built using our irida-plugin-builder - https://github.com/COMBAT-TB/irida-plugin-builder - which packs a tools.yaml inside them to help with tool-installation on Galaxy. we're working on an admin-tool to do tasks like install tools, start / stop the Workbench (i.e. bring docker-compose up / down), install singularity containers, run DMs, etc.
Aaron Petkau
@apetkau
Thanks so much. This looks great :)
I'll have to check it out
pvanheus
@pvanheus
(and yeah the Illumina pipeline is still iVar based... I need to get around to adding more robust variant calling like is in the latest covid-19-signal.... I think PHAC-NML is using a fork of that)
Aaron Petkau
@apetkau
:+1:
pvanheus
@pvanheus
while I've got you here - is there a technical reason that the "select" parameters aren't drop-downs? does IRIDA perhaps not know what the options of the Galaxy tool are?
Aaron Petkau
@apetkau
Yes, mainly that we never implemented that feature
Which isn't really a reason lol
So no technical issue for why it couldn't be
pvanheus
@pvanheus
ok, its a bit of a bummer for usability. I didn't spot an issue about it though, will need to create one
Aaron Petkau
@apetkau
Yes, IRIDA is not aware of what options of the Galaxy tool there are.
Yes, if you wanted to create an issue that would be great
pvanheus
@pvanheus
btw another thing I'm working on is updated DMs for the pangolin data. pangolin is a bit of a beast here... you either (1) download a fresh DB each time you run it or (2) use the DB installed with the tool (which gets stale quite quickly) or (3) choose data from a data table (but because the database changes so rapidly the data table will get very full very fast and there is no way to ensure that the most recently data table entry will be chosen by default)
to make (3) more usable will probably need changes on the Galaxy side. anyway. its a work in progress.
Aaron Petkau
@apetkau
That's awesome. But yes, sounds like something that needs to be dealt with. For option (3), it's possible to name the installed database based off of the date downloaded (e.g., pangolin-data-2022-05-24). Then IRIDA could sort the entries in the data-table by the name and pick the most recent as default.
I don't think IRIDA sorts the entries it displays from datatables in any particular order right now though
Or if there was a way to mark one data table entries as "current" and have it point to the most recently downloaded one. But that sounds like changes in Galaxy
Another option is to set something up behind-the-scenes which updates the database packaged with the tool (i.e., run pangolin --update in the conda environment), but that breaks reproducibility
pvanheus
@pvanheus
this is kind of related to the select question
Aaron Petkau
@apetkau
Yes.
pvanheus
@pvanheus
yeah running pangolin --update-data is option (1). which is also very inefficient when pipelines are run a single sample at a time (and potentially a flurry of pangolin --update-data commands run)
(1) is in fact the default behaviour of the Galaxy pangolin (and nextclade) tools (and the reason I added the current data updating code to pangolin)
Aaron Petkau
@apetkau
Oh, okay. I didn't realize that was the deafult.

this is kind of related to the select question

IRIDA should display databases derived from a Galaxy data table in a drop-down menu, though I don't think it's sorted in any particular order:

image.png
Karin Lagesen
@karinlag
@pvanheus and @apetkau We don't have good resource management
currently we are seeing that galaxy and slurm fubars on us quite regularly
thus we are thinking of asking the irida devs for a meeting, because we are obviously screwing something up :D
pvanheus
@pvanheus
@karinlag our deploy is a very basic one - all services on one server. I do have a few bugs with deleting projects but otherwise things work ok (once I have the Galaxy side working with software installs). but we are not in production mode yet - we will be bringing that online in the coming months and building up experience...
Karin Lagesen
@karinlag
we have split stuff out on different servers
we have ansible recipes for each of them too, if anyone is interested :)
Aaron Petkau
@apetkau

thus we are thinking of asking the irida devs for a meeting, because we are obviously screwing something up :D

Oh? I'm sorry to hear that. Yes, we would be happy to discuss further to figure out what's going on.

@pvanheus that's great news you will be bringing more resources online. Which bugs were they with deleting projects?

we have ansible recipes for each of them too, if anyone is interested :)

Yes, that would be awesome @karinlag :smile:

I wonder if we should have a section in our documentation that maybe points to other resources for installing/setting up IRIDA that people may have
pvanheus
@pvanheus

@pvanheus that's great news you will be bringing more resources online. Which bugs were they with deleting projects?

the error log is here - https://gist.github.com/pvanheus/dccb2ef42497e744c7a441f3bc1cb7e1- this is after deleting an analysis which was in Error state

btw is there any interest in a Hydra pipeline for IRIDA? you already have a Web Hydra resource...
8 replies
Aaron Petkau
@apetkau

@pvanheus that's great news you will be bringing more resources online. Which bugs were they with deleting projects?

the error log is here - https://gist.github.com/pvanheus/dccb2ef42497e744c7a441f3bc1cb7e1- this is after deleting an analysis which was in Error state

Thanks. Was this analysis shared with a project and you cannot delete the project without deleting the analysis?

I can reproduce the error, though I'm not sure exactly what is happening yet.

pvanheus
@pvanheus
The analysis was not shared. I didn't try and delete the project.
Aaron Petkau
@apetkau

Okay. I created an issue for it here phac-nml/irida#1287

For my test, the analysis submission was still deleted, but that exception was still thrown. Was this the case for you (as in the analysis submission was still deleted)?