These are chat archives for nextflow-io/nextflow

12th
Sep 2015
Paolo Di Tommaso
@pditommaso
Sep 12 2015 20:27
@robsyme it turns out there's also another git+torrent project
Robert Syme
@robsyme
Sep 12 2015 20:42
Nice find. I'm guessing that by defining it's own network protocols, it already works with nextflow. Does nextflow just shell out to git?
Paolo Di Tommaso
@pditommaso
Sep 12 2015 20:43
No. Nextflow uses JGit (a java based implementation)
Robert Syme
@robsyme
Sep 12 2015 20:52
Ah, and JGit faithfully implements the network protocols anyway :)
Paolo Di Tommaso
@pditommaso
Sep 12 2015 20:54
um, not sure but I guess no .. :)
however an option could be to shell out to plain git to handle gittorrent
I need to investigate a bit on that, it could be an interesting option to handle large bioinformatics dataset
Robert Syme
@robsyme
Sep 12 2015 21:00
One advantage of IFPS is that it can be just mounted. You could just ipfs add my_project_root and then share the returned hash.
This would give everybody access to all /previous/ commits, but for all subsequent commits, you'd need to re-ipfs add the project and re-share the hash, which is a bit of a pain. One solution would be to use the IPNS (..Naming Scheme as opposed to ..File System).
Archiving for publication is easy, as the publisher would just have to ipfs pin $PROJECT_HASH and then everything would be available with a stable ID forever.
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:03
looking at this example it seems that to share a git project it need it as a bare repository
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:21
BTW did you solve the #!/bin/bash --login problem ?
Robert Syme
@robsyme
Sep 12 2015 21:27
Hmm. Just testing the bare vs normal git now.
Actually, to sort out the --login issue, I just sourced the specific components I needed in .bashrc.
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:29
I see
Robert Syme
@robsyme
Sep 12 2015 21:38
Aha:
nextflow run -r bb93c7d5d0613caf4a70145e8c2211cf55a39a99 /ipfs/QmUrRaPgAVa1BYQThhLNEWP7FATUQrmb9pP1r3Kb3233QW
fails with
Revision option cannot be used running a local pipeline script
I didn't realise that you couldn't run particular revisions without github/bitbucket.
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:39
yep
it just read the file
but actually I could check if exists the .git folder ..
Robert Syme
@robsyme
Sep 12 2015 21:42
If people (me) really wanted to use IPFS, I could just set up a post-commit hook to push the new version up to IPNS.
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:42
so basically your idea is just share a plan git repository and access it using a IPFS mount, isn't it?
Robert Syme
@robsyme
Sep 12 2015 21:42
Yup
Then it could be nextflow run /ipns/robsyme.com/whatever/path/example-project/$COMMIT
That plan wouldn't ask anything extra of nextflow.
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:46
that's nice.
but nextflow should be able to handle a local git repo
Robert Syme
@robsyme
Sep 12 2015 21:46
That would also be great ;)
Paolo Di Tommaso
@pditommaso
Sep 12 2015 21:47
wait, otherwise I don't see how you can manage to run as you said before
I mean nextflow run /ipns/robsyme.com/whatever/path/example-project/$COMMIT
Robert Syme
@robsyme
Sep 12 2015 21:53
I didn't explain that very well.
I'll just set up an example to make sure it works, one sec.
Robert Syme
@robsyme
Sep 12 2015 22:16
I was thinking that for each commit, you create a new folder at a particular location and then example_hash=`ipfs add -r dir` everything together, then ipfs name publish $example_hash.
But that's getting a bit complicated. It would be great if IPFS looked for .git
Paolo Di Tommaso
@pditommaso
Sep 12 2015 22:18
I think the same ;)
I will give it a try, I will let you know
Robert Syme
@robsyme
Sep 12 2015 22:20
Excellent :)
Paolo Di Tommaso
@pditommaso
Sep 12 2015 22:22
time to go for me, bye
Robert Syme
@robsyme
Sep 12 2015 22:44
Catcha