Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 08 2018 16:17
    thejmazz edited #95
  • Jul 08 2018 04:50
    thejmazz edited #95
  • Jul 02 2018 17:24
    thejmazz opened #95
  • Jun 27 2018 16:45

    thejmazz on hotfix-paramsString-in-trajectory

    (compare)

  • Jun 27 2018 16:45

    thejmazz on dev

    remove paramsString from trajec… Merge pull request #90 from bio… (compare)

  • Jun 27 2018 16:45
    thejmazz closed #90
  • Jun 27 2018 16:44

    evoxtorm on dev

    Added test for workflow-status Merge pull request #94 from bio… (compare)

  • Jun 27 2018 16:44
    evoxtorm closed #94
  • Jun 27 2018 16:07

    evoxtorm on input-staging

    (compare)

  • Jun 27 2018 15:54
    thejmazz review_requested #90
  • Jun 27 2018 15:44

    evoxtorm on input-staging

    Input staging (compare)

  • Jun 27 2018 15:41

    evoxtorm on input-staging

    (compare)

  • Jun 22 2018 15:58
    thejmazz closed #93
  • Jun 22 2018 15:58
    thejmazz commented #93
  • Jun 22 2018 15:53
    codecov-io commented #94
  • Jun 22 2018 15:50
    evoxtorm opened #94
  • Jun 22 2018 15:46

    evoxtorm on workflow-status

    Added test for workflow-status (compare)

  • Jun 22 2018 15:22

    evoxtorm on workflow-status

    (compare)

  • Jun 22 2018 15:04
    codecov-io commented #93
  • Jun 22 2018 15:01
    evoxtorm synchronize #93
Bruno Vieira
@bmpvieira
Hey @thejmazz , @Lonewolfenrir created this pipeline in bash https://github.com/Lonewolfenrir/FunSec
it's still in an early stage, so maybe with some help he could easily convert it to watermill
Julian Mazzitelli
@thejmazz
quickly scanning through, to convert would essentially need to isolate commands
and all the mkdirs would go away ;)
João Baptista
@Lonewolfenrir
ya basically i need to create files so for the next program to read
needed*
Bruno Vieira
@bmpvieira
yup, seems like it's still small enough to be easily converted and maybe if he likes watermill, he could start converting the rest of the lab (CoBiG2.com)
Julian Mazzitelli
@thejmazz
essentially watermill lets you define input/output files via glob patterns, and then compose these steps together
João Baptista
@Lonewolfenrir
ahah then we could do a study group here
Julian Mazzitelli
@thejmazz
so you could touch a.foo as one task, declare *.foo as output, ahve next task declare *.foo as input, and it will handle everything in its own folder, and put proepr symlinks in

one big issue that needs to be fixed asap is that, if a previous task in the lineage is reran, since next onw just checks for file existence, it will be skipped.
I think i paritally added hashing for that, but theres still some stuff to be done

buit if piopeline is ran beginnign to end its fine
that issue is for iterative editing of specific steps and not hjaving to rerun all of them

João Baptista
@Lonewolfenrir
so the pipeline read multiple fasta files, each represents a proteome, then we pipeline executes various programs on each files. So we just need to make it easy to run pararell runs
Julian Mazzitelli
@thejmazz
is each run basically input -> outout -> input -> output -> .. in a linear chain?
with different tools
João Baptista
@Lonewolfenrir
exactly
the output of the first program would become the input for the second program and so on
Julian Mazzitelli
@thejmazz
and it should run that on each *.fasta (or whichever pattern), in parallel?
João Baptista
@Lonewolfenrir
yep
in the middle of the pipeline i needed to combine the output of two programs and only selected the proteins that were common in both outputs
Julian Mazzitelli
@thejmazz

ok - one thing I want to point out
is distinguish between parallelibility and generality

being able to run a pipeline on *.x files is "generality", (lets you define reusable tasks), and can be extended to parallel (basically Promise.map(tasks, t => t(), { concurrency: x }) from bluebird), but true parallel has other concerns like number of processes to be ran on system which watermill does not address

this will be a good example to see where to take watermill API, as i think its possible to do what you ask withj JS + watermill task objects

João Baptista
@Lonewolfenrir
ah okok ths
yes i want to run the pipeline on multiple files
Julian Mazzitelli
@thejmazz
if you can set up a basic linear pipeline with watermill, then we can take it from their to have it run that over multiple files
ill be happy to help out.
João Baptista
@Lonewolfenrir
Ths! i still need to learn the basics of javascript and node.js