Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 20 18:04

    bmpvieira on master

    Refactor to add CLI 3.0.0 (compare)

  • Nov 20 18:04

    bmpvieira on master

    Refactor to add CLI 3.0.0 (compare)

  • Sep 02 15:27
    dependabot[bot] labeled #6
  • Sep 02 15:27
    dependabot[bot] labeled #6
  • Sep 02 15:27
    dependabot[bot] opened #6
  • Sep 02 15:27

    dependabot[bot] on npm_and_yarn

    Bump bl from 1.2.2 to 4.0.3 Bu… (compare)

  • Sep 02 15:27
    dependabot[bot] opened #6
  • Sep 02 15:27

    dependabot[bot] on npm_and_yarn

    Bump bl from 1.2.2 to 4.0.3 Bu… (compare)

  • Jul 27 16:44
    JWhoops closed #5
  • Jul 27 16:44
    JWhoops closed #5
  • Apr 16 04:31
    JorgeFelixc opened #44
  • Apr 16 04:31
    JorgeFelixc opened #44
  • Nov 19 2019 17:27
    katrinleinweber commented #43
  • Nov 19 2019 17:27
    katrinleinweber commented #43
  • Nov 19 2019 17:27
    katrinleinweber commented #43
  • Nov 19 2019 17:27
    katrinleinweber commented #43
  • Nov 19 2019 13:28

    bmpvieira on master

    Hyperlink DOIs to preferred res… Merge pull request #5 from katr… (compare)

  • Nov 19 2019 13:28

    bmpvieira on master

    Hyperlink DOIs to preferred res… Merge pull request #5 from katr… (compare)

  • Nov 19 2019 13:28
    bmpvieira closed #5
  • Nov 19 2019 13:28
    bmpvieira closed #5
Hitesh Joshi
@evoxtorm
Thanks for the guidance @tiagofilipe12
Julian Mazzitelli
@thejmazz
Hitesh Joshi
@evoxtorm
hey I have question??? The bionode-watermill is purely made on nodejs and html or any other language is used in it!!!!
Julian Mazzitelli
@thejmazz
Yes its only javascript. But it is used to call CLI programs which can be in any language
Tiago Jesus
@tiagofilipe12
And html is just a quick visualization tool for pipeline workflow graph
Hitesh Joshi
@evoxtorm
thanks
cli- command line interface or anything else
Julian Mazzitelli
@thejmazz
Look at nodejs child_process, if you return a string from an operationCreator for a task in watermill it will use child_process.spawn with { shell: true } (so you can use unix pipes)
Otherwise it can return a function with callback, promise, or node stream (though I think the code that checks for this is WIP - and probably we should add an option to task for user to declare what their operationCreator returns)
So basically anything you can run from terminal
Julian Mazzitelli
@thejmazz
And the stdout, stdin, stderr of that process are made available as node streams by the child_process module
Julian Mazzitelli
@thejmazz

One cool thing to do would be to use ES6 template literals to do something like

const { task, python } = require('bionode-watermill')

const mytask = task({
  input: '*.txt',
  output: '*.foo',
  name: 'My task'
}, ({ input }) => python`
  with open('output.foo') as output:
      with open('${input}') as input:
          for line in input:
            output.write(line.upper())
`)

// where `python` returns something like
// { language: 'python', executer: '/usr/bin/env python3', code }
// watermill writes the script, marks it executable, stores it in task dir

then it would be frictionless to experiment with quick scripts in a bunch of languages, write small simple scripts for visualization with R, etc. using ${variable in source code for a script is kinda sketchy but maybe OK for quick experimentaion. Nextflow has something similar for scripts in any language (you just put #!/usr/bin/env python3 at top)

OR maybe that program should go under a prop of the task, so it is called as a CLI like usual

(excuse my python im not sure if its correct)
the operationCreator is the second paramater passed to task, the first is the object props
minoltafan
@minoltafan
hello, world =)
Hitesh Joshi
@evoxtorm
hey bionode-ncbi is a beautiful library and I used this library in my project
and I'm thinking I will give a small presentation on bionode-ncbi telling other guys how to use it
My professors are loving this library
Bruno Vieira
@bmpvieira
Hey @evoxtorm ! Super happy to read that! :smile:
Thanks for using it! Would love to see those slides and know more about how you are using bionode-ncbi in your work
Hitesh Joshi
@evoxtorm
ok sure sure I'll share with you
Hitesh Joshi
@evoxtorm
Hey guyz,
Does anyone have data flow diagram of bionode-ncbi because I have to show how this library actually works
Julian Mazzitelli
@thejmazz
its mostly filtering/wrapping ncbi API calls, making a stream out of paginated response s, etc, there is no diagram I am aware of. is this use case being able to explain underlying technicalities so that others feel more comfortable with the software?
Hitesh Joshi
@evoxtorm
Ok Thanks @thejmazz
Olga Botvinnik
@olgabot
hello! I'm trying to download FASTQ files from this bioproject: PRJNA183192
But bionode-ncbi download sra PRJNA183192 repeats the first SRA file, SRR629557.sra until infinity
Bruno Vieira
@bmpvieira
Hey @olgabot ! What do you get if you just do bionode-ncbi urls sra PRJNA183192
also, bionode-ncbi download is very verbose by default (to pipe into other tools)
you can try bionode-ncbi download sra --pretty PRJNA183192 to display a progress bar instead
Bruno Vieira
@bmpvieira
if for some reasonbionode-ncbi urls gives you the right urls, but bionode-ncbi download keeps downloading the same file, please open an issue on GitHub and try this as a workaround: bionode-ncbi urls sra PRJNA183192 | jq .url | xargs wget
or if you don't have jq installed: bionode-ncbi urls sra PRJNA183192 | sed 's|.*url":"\(.*\)","uid.*|\1|' | xargs wget
Hitesh Joshi
@evoxtorm
Hey this year also bionode has same project
bionode-watermill
Hitesh Joshi
@evoxtorm
In GSOC
Julian Mazzitelli
@thejmazz
yes! i will try to make a more detailed project description soon!
Hitesh Joshi
@evoxtorm
ok
Maximilien Rothier Bautzer
@cachemoi
Hi,
Does bionode already have a module focused on systems biology? I can't seem to find one. If it hasn't been done already and if it's a feature you're looking for i'd like to start working on it.
As a simple first feature i'm thinking an SBML/JSON parser might be interesting
Julian Mazzitelli
@thejmazz
everything we have is a repo on bionode organization. no SBML parser atm
nice!
Sarthak Sehgal
@sarthak-sehgal
Hello @all
I'm Sarthak from India
I'm interested in the "Bionode workflow engine for streamed data analysis (bionode-watermill)" project for GSoC 2018
How do I get started? :smiley:
Julian Mazzitelli
@thejmazz
Hi Sarthak! These are some points of interest:
  • playing with bioinformatics pipelines using shell scripts which clearly demonstrates difficulty (see my ngs-worklfows blog post on jmazz.me)
  • taking a look at the CWL project, and corresponding ecosystem (rabix team on GitHub for example), similarly WDL (workflow standard the Broad institute made)
  • reading some review literature on variant calling perhaps (the majority of pipelines are this, producing variant calling files (VCF) as output, for human genomics) (I would not expect students to fully understand these papers but I should look for a good introduction one)
  • the biological intuition probably would take a while to really get (but not the technical implementation of tools as much perhaps, which isn't required to run them...), at a superficial level you are calling a bunch of command line tools, each with their own set of parameters (understanding how these parameters affect the final output and the biological implications of such is the crux of being informed technically as well as biologically). in this way a pipeline tool would benefit from being able to assign a matrix of commands/parameters and then compare outputs (goal of fork() in watermill)
Let me know if anything unclear
Sarthak Sehgal
@sarthak-sehgal
Thanks a lot @thejmazz! I will start looking into them. :smiley:
Hitesh Joshi
@evoxtorm
Hey anyone have a sample for proposal
Julian Mazzitelli
@thejmazz

heres mine from 2016 Bionode workflow engine for streamed data analysis

note the timeline ended up very different from what proposed. but its good to get an idea if student can organize a plan

Hitesh Joshi
@evoxtorm
thanks