These are chat archives for nextflow-io/nextflow

21st
Sep 2017
Luca Cozzuto
@lucacozzuto
Sep 21 2017 07:35
does it run in parallel? :joy: :joy: :joy: :joy:
Paolo Di Tommaso
@pditommaso
Sep 21 2017 07:36
we should ask
Paolo Di Tommaso
@pditommaso
Sep 21 2017 08:40
I've just uploaded 0.26.0-SNAPSHOT including the following issues
you may want to give it a try adding the following var in your env NXF_VER=0.26.0-SNAPSHOT then using NF as usual
Anthony Underwood
@aunderwo
Sep 21 2017 08:48

@pditommaso

you may want to give it a try adding the following var in your env NXF_VER=0.26.0-SNAPSHOT then using NF as usual

worked for me with a workflow submitting to SGE and using a Singularity image

Paolo Di Tommaso
@pditommaso
Sep 21 2017 08:48
:ok_hand:
Simone Baffelli
@baffelli
Sep 21 2017 09:28
Not on a car but similar: bike
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:29
@skptic now, it your turn! :D
Evan Floden
@evanfloden
Sep 21 2017 09:30
I ride my bike to (mostly) free my mind from work :laughing:
Simone Baffelli
@baffelli
Sep 21 2017 09:30
I do it too
I ride to work as well, when the weather is not too severe
Evan Floden
@evanfloden
Sep 21 2017 09:31
@baffelli If you are ever in Barcelona and want to ride, I’m out most weekends (and weekdays commuting)
Simone Baffelli
@baffelli
Sep 21 2017 09:31
Would love to! I'm considering to go for a long bike tour when I'm done here
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:31
@skptic there are better ways :)
Simone Baffelli
@baffelli
Sep 21 2017 09:32
perhaps crossing France and Spain from east to west
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:32
you are crazy guys ..
Simone Baffelli
@baffelli
Sep 21 2017 09:32
Why?
It is addictive, just try
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:33
I could join with this
Simone Baffelli
@baffelli
Sep 21 2017 09:33
that's cheating
:grin:
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:33
who cares! :D
being one with the machine
Simone Baffelli
@baffelli
Sep 21 2017 09:34
One should follow the rules : http://www.velominati.com/the-rules/
And if you don't suffer a bit, it is not fun
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:35
that's for me, you can suffer as much as you want :joy:
Simone Baffelli
@baffelli
Sep 21 2017 09:36
Plus, if you cycle a lot you get to eat a lot :yum:
Phil Ewels
@ewels
Sep 21 2017 09:37
haha, yes! +1 for velominati link
(# 12 being the classic and most repeated one of the lot I think)
Simone Baffelli
@baffelli
Sep 21 2017 09:38
Indeed
I'm not playing by the rules actually

Especially

Mountain bike shoes and pedals have their place

Because I'm using SPDs only
Phil Ewels
@ewels
Sep 21 2017 09:45
much easier to walk in..!
Simone Baffelli
@baffelli
Sep 21 2017 09:48
True
But its getting cold here :worried:
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:49
waiting you all here
Simone Baffelli
@baffelli
Sep 21 2017 09:50
Well, just need a job there
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:51
last of the problems ..
Simone Baffelli
@baffelli
Sep 21 2017 09:54
It looks beautiful, for cycling as well
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:56
actually @thejmazz can say more about cycling in barcelona
Luca Cozzuto
@lucacozzuto
Sep 21 2017 09:57
I am lazy :)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:57
much better in front a big pizza :D
Simone Baffelli
@baffelli
Sep 21 2017 09:58
after a big ride I always like a big pizza or two
Luca Cozzuto
@lucacozzuto
Sep 21 2017 09:58
patin.jpg
Paolo Di Tommaso
@pditommaso
Sep 21 2017 09:58
ahahahahha
I can't believe, the NF branded vehicles series :joy:
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:09
@ewels is your plan to include all the fields in the report?
Phil Ewels
@ewels
Sep 21 2017 10:12
Yes! In fact I was thinking of removing that function and doing the sanitisation in the template file itself
that code is there from when I was trying to replicate exactly what you were doing in the original timline report code that I was basing this on
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:13
I see, but it still needed to wrap string value, isn't it?
Phil Ewels
@ewels
Sep 21 2017 10:14
how do you mean?
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:15
I'm not understanding how you can sanitize it in the template
if I'm not wrong that code generates the json, hence string values need to be wrapped and escaped
Phil Ewels
@ewels
Sep 21 2017 10:16
it doesn't really create json as such, that's done with the variables in the template here
but yes, the variables need to be escaped at some point - but probably cleaner / simpler if that's done in the above template file as well
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:17
I see
Phil Ewels
@ewels
Sep 21 2017 10:17
My overall aim was to try to move all representation of data into the template file and keep the groovy core file as generic as possible
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:17
I agree
Phil Ewels
@ewels
Sep 21 2017 10:18
that would then make it easier for multiple template files to be written which do different things with the data in whatever way that they want
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:19
let me work a bit on that
Phil Ewels
@ewels
Sep 21 2017 10:19
so yeah, the template file should probably instead have this instead then:
"task_id": "${StringEscapeUtils.escapeJavaScript(v.task_id)}",
(etc)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 10:20
here you mean ?
Phil Ewels
@ewels
Sep 21 2017 11:23
yup!
Francesco Strozzi
@fstrozzi
Sep 21 2017 12:51

guys one question. I am looking at the RNA-seq example pipeline https://github.com/nextflow-io/rnaseq-nf/blob/master/main.nf

In this case the index process store the index files into an output channel which is the input of the “quant” process. So once a single task is run then the channel storing the index will be empty, so this example will only work for 1 single sample. Is that correct or am I missing something ? When we need to use index files for each task, we declare them in the input of a process as file using some params and we avoid using Channels since those will be consumed.

just curious to know what NF users are doing in these cases
Luca Cozzuto
@lucacozzuto
Sep 21 2017 12:52
not the single task but when every task from that channel is finished
the process quant runs on every file (reads)
Francesco Strozzi
@fstrozzi
Sep 21 2017 12:55
so the channel storing the index will not be consumed until all the task are completed ?
Luca Cozzuto
@lucacozzuto
Sep 21 2017 12:55
and once every task is finished you consume the channel index_ch
all the tasks from the process quant
@pditommaso correct me if I am wrong :)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 12:58
this is a tricky aspect and top hit question in our *virtual* FAQ
the point is that since the index process can be execute only one because it gets only one file
the output channel index_ch is implicitly a singleton channel
that means you can read as many time as you want but it always returns the same value
therefore when combine a singleton channel with a consumable channel like here
the cardinality of executions is given by the number of entries if the latter ie read_pairs_ch
in practice you get always the same index file combine with different read pairs
.
.
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:03
does it make sense ? :D
Luca Cozzuto
@lucacozzuto
Sep 21 2017 13:05
paolo I'm waiting for you
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:05
why ?
Luca Cozzuto
@lucacozzuto
Sep 21 2017 13:05
wait
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:05
what if I have an index with multiple files (which is the common case for NGS aligners for instance), in that scenario if I push in an output channel all the index files doing something like file(“human_*") does all this holds true ?
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:06
yes
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:06
because in that case the ouput channel will hold a single element, which is a list of files ?
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:06
exactly
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:07
:smile:
Luca Cozzuto
@lucacozzuto
Sep 21 2017 13:07
next_coffe.jpg
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:07
nooooooo
me too
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:07
:clap:
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:08
thanks paolo and luca for the explanation
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:08
welcome
Luca Cozzuto
@lucacozzuto
Sep 21 2017 13:08
welcome
if you want multiple indexes you can use combine()
to combine the channels
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:17
IMG_20170921_151315.jpg
not the same thing :(
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:18
definitely not!
Simone Baffelli
@baffelli
Sep 21 2017 13:54
@lucacozzuto that is a proper cofee machine. Non that capsule-based abomination that @fstrozzi posted
Francesco Strozzi
@fstrozzi
Sep 21 2017 13:54
:(
I know
Simone Baffelli
@baffelli
Sep 21 2017 13:54
Fortunately we have a good one upstairs
IMG_20170921_112340.jpg
let's continue the series "nextflow stickers in unlikely places"
Paolo Di Tommaso
@pditommaso
Sep 21 2017 13:55
:+1: :+1:
Félix C. Morency
@fmorency
Sep 21 2017 14:19
This message was deleted
ashkurti
@ashkurti
Sep 21 2017 14:39
hello, I am just starting to see if I can use nextflow to build up pipelines at my work.
by using for example the lsf executor
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:40
welcome on board
ashkurti
@ashkurti
Sep 21 2017 14:40
Thanks - I might have a few questions in the next few days :)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:40
obviously :)
ashkurti
@ashkurti
Sep 21 2017 14:41
in case an executable takes some implicit inputs and needs them all at once
is it sufficient to specify the inputs at the start of the process
as for example input: file('./BTNOL/*')
if so, after my nextflow script fails, I would at least expect to find symbolic links of these input files at the working directory
but I have not found those there
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:44
have you set a scratch directory for the proc execution ?
ashkurti
@ashkurti
Sep 21 2017 14:45
nope
how can I do this
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:45
can you share you code or at least the problematic part ?
ashkurti
@ashkurti
Sep 21 2017 14:45
sure
#!/usr/bin/env nextflow

process sims {
    input:
    file('./BTNOL/*')

    output:
    file 'OUTPUT'

    cpus 24
    executor 'lsf'
    time '1h'
    beforeScript '. /etc/profile.d/modules.sh', 'module load intel_mpi/5.0.3'
    afterScript 'touch complete'

    """
    mpiexec.hydra -np 24 path_to_sims.exe
    """
}
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:48
trick number one: wrap your code in ``` new-line CODE new-line ```
GOOD
uhh mpiexec ..
ashkurti
@ashkurti
Sep 21 2017 14:49
:)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:49
:)
I think the problem is here
  input:
    file('./BTNOL/*')
basically you want to pass all the files in the folder BTNOL ?
ashkurti
@ashkurti
Sep 21 2017 14:50
ok so that folder contains 3 files and all 3 files are implicit inputs for the .exe
yes
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:51
if you only need to provide as input the directory
(a directory is a file ..)
use this
ashkurti
@ashkurti
Sep 21 2017 14:52
ok without the wildcard
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:52
   input:
   file btnol_dir from file('./BTNOL')
wait
how does hydra access those files? I don't see any file name ?
ashkurti
@ashkurti
Sep 21 2017 14:54
that s why I was referring to them as implicit inputs
basically they need to be found in the same directory from which the script is being launched
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:56
let's put in this way
ashkurti
@ashkurti
Sep 21 2017 14:57
I think these pose a problem in this case as those files need to be transferred wherever the script generated by nextflow will be launched from
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:57
with my snippet in the task work directory, there will be a folder (symlink) named BTNOL
is that fine or do you need each single file in that folder staged in the task work dir ?
ashkurti
@ashkurti
Sep 21 2017 14:58
yes each single file
Paolo Di Tommaso
@pditommaso
Sep 21 2017 14:58
ookk
if so, you need this
ashkurti
@ashkurti
Sep 21 2017 15:00
when nextflow fails that points to work dir and in that work dir I don't see a sym link even after using your snippet
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:00
input: 
 file btnol_dir from Channel.value(file('./BTNOL/*') )
ashkurti
@ashkurti
Sep 21 2017 15:00
drwxr-xr-x 2 user user  512 Sep 21 15:55 .
drwxr-xr-x 3 user user  512 Sep 21 15:55 ..
-rw-r--r-- 1 user user    0 Sep 21 15:55 .command.begin
-rw-r--r-- 1 user user 2949 Sep 21 15:55 .command.log
-rw-r--r-- 1 user user 2580 Sep 21 15:55 .command.run
-rw-r--r-- 1 user user  132 Sep 21 15:55 .command.sh
-rw-r--r-- 1 user user    3 Sep 21 15:55 .exitcode
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:00
the above snippet should produce the expected inputs
ashkurti
@ashkurti
Sep 21 2017 15:00
ok thanks I will try the above
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:00
if it works I will explain you the rationale behind it :)
ashkurti
@ashkurti
Sep 21 2017 15:03
it still fails.
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:03
:(
ashkurti
@ashkurti
Sep 21 2017 15:03
and I still cannot see any symlinks to the files it expects in input
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:03
what's the content of the workdir ?
ashkurti
@ashkurti
Sep 21 2017 15:05
drwxr-xr-x 2 user user  512 Sep 21 16:03 .
drwxr-xr-x 3 user user  512 Sep 21 16:01 ..
-rw-r--r-- 1 user user    0 Sep 21 16:01 .command.begin
-rw-r--r-- 1 user user 2949 Sep 21 16:02 .command.log
-rw-r--r-- 1 user user 2811 Sep 21 16:01 .command.run
-rw-r--r-- 1 user user  132 Sep 21 16:01 .command.sh
-rw-r--r-- 1 user user    3 Sep 21 16:01 .exitcode
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:06
let me try
ashkurti
@ashkurti
Sep 21 2017 15:08
ok thanks Paolo
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:08
wait, I still need to answer ;)
ashkurti
@ashkurti
Sep 21 2017 15:08
I have another thanks saved for when you answer
:)
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:09
it works ..
process foo {
input: 
 file btnol_dir from Channel.value(file('./BTNOL/*') )

 '''
 ls -la 
 '''
}
I have the following
-rw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.begin
-rw-r--r--   1 pditommaso  staff    60 Sep 21 17:09 .command.env
-rw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.err
-rw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.log
-rw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.out
prw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.pe
prw-r--r--   1 pditommaso  staff     0 Sep 21 17:09 .command.po
-rw-r--r--   1 pditommaso  staff  2062 Sep 21 17:09 .command.run
-rw-r--r--   1 pditommaso  staff    23 Sep 21 17:09 .command.sh
lrwxr-xr-x   1 pditommaso  staff    43 Sep 21 17:09 A -> /Users/pditommaso/projects/nextflow/BTNOL/A
lrwxr-xr-x   1 pditommaso  staff    43 Sep 21 17:09 B -> /Users/pditommaso/projects/nextflow/BTNOL/B
lrwxr-xr-x   1 pditommaso  staff    43 Sep 21 17:09 C -> /Users/pditommaso/projects/nextflow/BTNOL/C
sorry need to leave now
ashkurti
@ashkurti
Sep 21 2017 15:12
ok was trying your same script
when you can it would be useful to know how do you find out which is the working directory if it does not print that out
thanks for helping on this
Paolo Di Tommaso
@pditommaso
Sep 21 2017 15:16
The prefix of the prefix is the same as the task ID printed on the screen
Use tab completion etc voilà..
*et
Simone Baffelli
@baffelli
Sep 21 2017 15:30
or make the process deliberately fail
and copy the path from the error message :grin: i
ashkurti
@ashkurti
Sep 21 2017 15:33
thanks both! I can now see that the simple script with a local executor has created the links to the input files
what if I want to have a specific output file transferred to the launching directory
for example I tried this:
#!/usr/bin/env nextflow

process foo {
    input:
    file btnol_dir from Channel.value(file('./BTNOL/*'))

    output:
    file "out_file"

    """
    ls -la > out_file
    """
}
And out_file is not present at the launching directory although it is present at the work dir
Simone Baffelli
@baffelli
Sep 21 2017 15:36
what do you mean with launching directory?
ashkurti
@ashkurti
Sep 21 2017 15:37
with launching directory I meant directory from which we launch the nextflow workflow
Simone Baffelli
@baffelli
Sep 21 2017 15:37
files are only present in the work dir and then passed to processes via channels, you can then copy the outputs somehwere using publishDir
but by default files are only kept in the workdirs of each process
ashkurti
@ashkurti
Sep 21 2017 15:38
ok thanks for the info
Simone Baffelli
@baffelli
Sep 21 2017 15:39
so you could use

#!/usr/bin/env nextflow
process foo {
    publishDir: "launchingDir"
    input:
    file btnol_dir from Channel.value(file('./BTNOL/*'))

    output:
    file "out_file"

    """
    ls -la > out_file
    """
}
and you will see out_file in "launchingDir"
ashkurti
@ashkurti
Sep 21 2017 15:40
thanks for elaborating further as I was unseccessfully looking for the publishDir directive
ashkurti
@ashkurti
Sep 21 2017 15:43
ok thanks
Julian Mazzitelli
@thejmazz
Sep 21 2017 17:43
@baffelli @pditommaso aha yes amazing mountain biking up in parc de collserola! first ride I went west along ridge (great city views) and second ride went more north up into the national park which was really nice. https://www.strava.com/activities/1019261166 && https://www.strava.com/activities/1022516886
Simone Baffelli
@baffelli
Sep 21 2017 19:17
Nice! But that bike...to many sospensions :grin:
Simone Baffelli
@baffelli
Sep 21 2017 19:22
I see that I'm not the only one that likes climbing by bike
Mike Smoot
@mes5k
Sep 21 2017 19:31
If we're all mountain bikers then maybe the next nextflow meeting can be in Moab... :)
Simone Baffelli
@baffelli
Sep 21 2017 19:34
I ride a cx bike, is that enough?
Otherwise i can offer a rigid mtb😂
Mike Smoot
@mes5k
Sep 21 2017 19:38
Well you can ride a rigid bike in Moab, I'm just not sure you'd want to! But don't worry, they rent really nice bikes there. :)
fmorency @fmorency also mountain biker!
Félix C. Morency
@fmorency
Sep 21 2017 19:39
I started this year! 4 rides this summer.
Simone Baffelli
@baffelli
Sep 21 2017 19:40
I propose switzerland
Good for both mtbers and roadies
And also for the weirdos like me that like riding skinny tyres on gravel roads
Mike Smoot
@mes5k
Sep 21 2017 19:41
Happy to compromise on Switzerland!
Simone Baffelli
@baffelli
Sep 21 2017 19:42
Also italy is close..all the famous passes
And
Mike Smoot
@mes5k
Sep 21 2017 21:19

hi @pditommaso I'm running into a problem with 0.25.6 where I'm running my pipeline with a tag: nextflow run <repo-url> -hub gitlab -r 1.0.1 -latest ... and this works the first time I run it, but fails on subsequent run on the same repo with this error:

Sep-21 21:15:14.495 [main] DEBUG nextflow.scm.AssetManager - Pull pipeline SGI-Pipelines/nextflow_test  -- Using local path: /tools/nextflow/assets/SGI-Pipelines/nextflow_test
Sep-21 21:15:15.532 [main] DEBUG nextflow.cli.Launcher - Operation aborted
org.eclipse.jgit.api.errors.DetachedHeadException: HEAD is detached
    at org.eclipse.jgit.api.PullCommand.call(PullCommand.java:202)
    at nextflow.scm.AssetManager.download(AssetManager.groovy:574)
    at nextflow.scm.AssetManager.download(AssetManager.groovy)
    at nextflow.cli.CmdRun.getScriptFile(CmdRun.groovy:280)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:197)
    at nextflow.cli.Launcher.run(Launcher.groovy:410)
    at nextflow.cli.Launcher.main(Launcher.groovy:564)

Is there a conflict with the -latest flag and the fact that I'm using a tag instead of a branch? This works fine with branches.

Paolo Di Tommaso
@pditommaso
Sep 21 2017 21:27
#389 ?
are you able to replicate it? still unable to have a test case to debug and fix.
Mike Smoot
@mes5k
Sep 21 2017 21:50
I'm definitely replicating it right now! I've very suspicious that this is related to using tag instead of a branch. I'll try to put together a test case and attach it to that ticket. Thanks!