These are chat archives for nextflow-io/nextflow

3rd
May 2018
Simone Baffelli
@baffelli
May 03 2018 06:40

in my experience so far, I've had to use def inside Channel closures, and could not use it inside Processes

Same experience here.

Simone Baffelli
@baffelli
May 03 2018 06:53
Perhaps because variables inside processes are already scoped? I presume NF uses lexical scoping, thus variables defined inside of a process are limited to that process?
gawells
@gawells
May 03 2018 11:30
What are possible causes for publishDir being ignored? It's failing in one of my processes, but apart from the script section being empty (the outputs are a result of collectFile) there's no major difference with the other processes
Paolo Di Tommaso
@pditommaso
May 03 2018 11:50
apart from the script section being empty
what do you mean ?
gawells
@gawells
May 03 2018 11:58
I don't have anything between ''' ''''
but I've tried putting in some dummy commands too, like touch. Still no directory with outputs
Paolo Di Tommaso
@pditommaso
May 03 2018 12:02
um, if there's no script, there's no output, therefore nothing is produced by publishDir ..
gawells
@gawells
May 03 2018 12:02
there is from the collectFile operation, I'm just not doing anything to the resulting files yet
Paolo Di Tommaso
@pditommaso
May 03 2018 12:03
publishDir publishes only the files that are created an output of that process
you should move it to the upstream process
gawells
@gawells
May 03 2018 12:06
so just specifying in input: and output: isn't sufficient? All I need is the result of collectFile. I tried running touch on the expected outputs too, but still nothing
Evan Floden
@evanfloden
May 03 2018 12:07
Can you provide your process?
gawells
@gawells
May 03 2018 12:08
process reGroupCount {

    input:
        file count_file from count.combine(group).collectFile() {item -> inGroup(item[0],item[1]) }


    output:
        file "${group}.combined.txt" optional true into grouped_files

    publishDir "groupings", mode: 'copy', pattern: '*txt', overwrite: true

    """

    """
}
the *.combined.txt are being produced in the work dirs
Paolo Di Tommaso
@pditommaso
May 03 2018 12:11
you should not use a process for that
the following code should do that
 count
  .combine(group)
  .collectFile() {item -> inGroup(item[0],item[1]) }
  .tap { grouped_files }
  .filter { file -> file.name.endsWith('.txt') }
  .subscribe { file -> file.copyTo("groupings") }
gawells
@gawells
May 03 2018 12:14
will try it, thanks
Paolo Di Tommaso
@pditommaso
May 03 2018 12:14
it's beautiful, isn't it :)
gawells
@gawells
May 03 2018 12:15
I don't recall seeing tap and filter in the docs yet
Paolo Di Tommaso
@pditommaso
May 03 2018 12:16
I think, you need to make sure the groupings directory exist before
filteris quite clear what's doing
tap is a bit more tricky, basically is a fork
gawells
@gawells
May 03 2018 12:17
yes, it works :)
Paolo Di Tommaso
@pditommaso
May 03 2018 12:17
:clap:
Shellfishgene
@Shellfishgene
May 03 2018 12:27
Is there a directive that controls the maximum number of jobs that are submitted to an executor?
marchoeppner
@marchoeppner
May 03 2018 12:27
queueSize=
Shellfishgene
@Shellfishgene
May 03 2018 12:31
Danke!
Sendu Bala
@sb10
May 03 2018 12:54

Hi, new to nextflow and groovy. Want to try writing a new executor. Stuck on make compile because `Downloading https://services.gradle.org/distributions/gradle-4.5.1-all.zip

Exception in thread "main" java.net.ConnectException: Connection timed out` I suspect it isn't paying attention to my https_proxy. What can I do?

Paolo Di Tommaso
@pditommaso
May 03 2018 12:57
try this
Sendu Bala
@sb10
May 03 2018 13:03
Thanks; would I add properties to gradle/wrapper/gradle-wrapper.properties?
That didn't work. Which is "the build script"? build.gradle?
Paolo Di Tommaso
@pditommaso
May 03 2018 13:09
nope, in add a gradle.properties file in project root dir
Sendu Bala
@sb10
May 03 2018 13:10
Ah, thanks. that worked.
Paolo Di Tommaso
@pditommaso
May 03 2018 13:14
@sven1103 thanks a lot for the PR, I will review asap
Sven F.
@sven1103
May 03 2018 13:23
@pditommaso of course, this was long overdue! Looking forward to your review :)
Hitesh Joshi
@evoxtorm
May 03 2018 13:49
Screenshot from 2018-05-03 19-17-02.png
Paolo Di Tommaso
@pditommaso
May 03 2018 13:50
please make sure Java 8 is installed
Hitesh Joshi
@evoxtorm
May 03 2018 13:50
In docs it is written java higher version also work
Nextflow can be used on any POSIX compatible system (Linux, Solaris, OS X, etc). It requires BASH and Java 8 or higher to be installed.
Nextflow can be used on any POSIX compatible system (Linux, Solaris, OS X, etc). It requires BASH and Java 8 or higher to be installed.
So that's mean it only run on java 8
Paolo Di Tommaso
@pditommaso
May 03 2018 13:52
Yes
where is written that stuff ?
Hitesh Joshi
@evoxtorm
May 03 2018 13:52
In requirements of main docs
Paolo Di Tommaso
@pditommaso
May 03 2018 13:53
you are right, need to update that
Hitesh Joshi
@evoxtorm
May 03 2018 13:54
Ok sure
Paolo Di Tommaso
@pditommaso
May 03 2018 13:54
latests java releases changed a lot and there are some incompatibilities in third parties libraries to solve
Hitesh Joshi
@evoxtorm
May 03 2018 13:55
ahhh again uninstall all the things
Paolo Di Tommaso
@pditommaso
May 03 2018 13:55
do you have docker engine ?
Hitesh Joshi
@evoxtorm
May 03 2018 13:56
Yes
Paolo Di Tommaso
@pditommaso
May 03 2018 13:56
so docker run nextflow/nextflow:latest nextflow info
Hitesh Joshi
@evoxtorm
May 03 2018 13:56
Ok sure
Thanks
Paolo Di Tommaso
@pditommaso
May 03 2018 13:58
you can even download only the wrapper ie curl get.nextflow.io > nextflow
then run it as nextflow -d info or nextflow -d run ..
Hitesh Joshi
@evoxtorm
May 03 2018 14:00
Hey how can run my own file in docker or edited file
I'm using it for the first time
Paolo Di Tommaso
@pditommaso
May 03 2018 14:01
how can run my own file
what file? nextflow script file?
Hitesh Joshi
@evoxtorm
May 03 2018 14:01
Yeah to be specific
Paolo Di Tommaso
@pditommaso
May 03 2018 14:02
nextflow -d run my_script_file.nf
Hitesh Joshi
@evoxtorm
May 03 2018 14:03
ohhh It can be run from where file is present
Paolo Di Tommaso
@pditommaso
May 03 2018 14:04
yes, there's a lot of magic docker file mounts
Hitesh Joshi
@evoxtorm
May 03 2018 14:04
Ok
Paolo Di Tommaso
@pditommaso
May 03 2018 14:05
native Java is still the better and more stable solution
Hitesh Joshi
@evoxtorm
May 03 2018 14:05
ok
arontommi
@arontommi
May 03 2018 14:10
Hi!
when does nextflow create a symlink in work directory and when does it create a hard link to files ?
Paolo Di Tommaso
@pditommaso
May 03 2018 14:12
symlinks are created by the task wrapper script, so the your task is executed
hard link to files (to published) file when the task is completed
arontommi
@arontommi
May 03 2018 14:12
okay, asking because my work dir is filling up with bam files that i used as input
Paolo Di Tommaso
@pditommaso
May 03 2018 14:13
that's good or bad ?
arontommi
@arontommi
May 03 2018 14:13
bad
Paolo Di Tommaso
@pditommaso
May 03 2018 14:13
why ?
arontommi
@arontommi
May 03 2018 14:13
because i want symlinks to them
i might not be explaining myself as well as i would like

so i have wgs bam files that are already aligned in backup project on a cluster,
and i want to run the through a pipeline(sarek) calling somatic variants,
i do that on a nobackup project:

so the first thing is i create symlinks,
then i run the pipeline and everything works great, but for some reason the pipeline copies the whole bam file into the work dir (for each process) instead of creating symlinks to the files

Paolo Di Tommaso
@pditommaso
May 03 2018 14:19
it could be the sarek was configured to copy that files instead of creating the symlinks
have tried to contact the sarek authors, namely @MaxUlysse ?
arontommi
@arontommi
May 03 2018 14:20
yea wanted to give him a break ;) was wondering if there was some easy solution, will look at the code
Paolo Di Tommaso
@pditommaso
May 03 2018 14:21
well, NF should creates a symlink for each input file unless specified otherwise
arontommi
@arontommi
May 03 2018 14:22
ok thanks
arontommi
@arontommi
May 03 2018 14:46
would using use PublishDir mode: link instead of copy result in this ? or does that only effect the publish dir ?
arontommi
@arontommi
May 03 2018 14:53
no wait! it only happens when the file is in output
Hitesh Joshi
@evoxtorm
May 03 2018 14:54
Screenshot from 2018-05-03 20-24-01.png
Hey this is it
Paolo Di Tommaso
@pditommaso
May 03 2018 14:55
not understanding your question
(BTW copy & paste works also with plain text .. :))
Hitesh Joshi
@evoxtorm
May 03 2018 14:56
I mean this is correct installation message
Paolo Di Tommaso
@pditommaso
May 03 2018 14:56
yes
Hitesh Joshi
@evoxtorm
May 03 2018 14:56
ohhh Thank god
The problem is due to java openjdk
Paolo Di Tommaso
@pditommaso
May 03 2018 14:57
looks strange
first time ever I've seen this problem
Hitesh Joshi
@evoxtorm
May 03 2018 14:57
hahahah In my case most thing happen first time
Can I close the issue
Paolo Di Tommaso
@pditommaso
May 03 2018 14:58
sure thing, maybe explain briefly how you solved it
Hitesh Joshi
@evoxtorm
May 03 2018 15:00
Ok I does nothing just uninstalled openjdk and jre then reinstalled it
Paolo Di Tommaso
@pditommaso
May 03 2018 15:00
ok, that's the equivalent of turning off and on again the computer :)
Hitesh Joshi
@evoxtorm
May 03 2018 15:02
hhahaha may I guess hahhaa but spend almost a day in doing this
what does this mean "you may complete the installation by moving it to a directory in your $PATH"
arontommi
@arontommi
May 03 2018 15:05
I think i have found my problem:
so what is the difference between doing file(bam) and file("$bam") in output
Paolo Di Tommaso
@pditommaso
May 03 2018 15:08
oh
the first stage the input file and assign to the bam variable the name of that file
arontommi
@arontommi
May 03 2018 15:09
yes!
so when manta was used to pipe the bam files to strelka, it piped the file and not the variable!
Paolo Di Tommaso
@pditommaso
May 03 2018 15:10
the second stage input file using the name provide by the bam variable
arontommi
@arontommi
May 03 2018 15:10
resulting in massive work fiels
yes
Paolo Di Tommaso
@pditommaso
May 03 2018 15:10
can you give me a link to the code ?
Paolo Di Tommaso
@pditommaso
May 03 2018 15:11
do you mean this line ?
arontommi
@arontommi
May 03 2018 15:12
yes
nice trick
Paolo Di Tommaso
@pditommaso
May 03 2018 15:13
it seems fine to me
arontommi
@arontommi
May 03 2018 15:13
sorry i ment the output
seenstevo
@seenstevo
May 03 2018 15:15
I am new to groovy and nextflow and am wondering how I might "export" a variable created in a bash script. such as: genome_size=faSize ${genome} | head -1 | cut -d " " -f 5. How might I get the variable $genome_size out into a channel? Thanks
Paolo Di Tommaso
@pditommaso
May 03 2018 15:16
@arontommi still I don't see the problem
@seenstevo can't, send the output to a file and post-process it
seenstevo
@seenstevo
May 03 2018 15:18
ok, will try that. thanks
arontommi
@arontommi
May 03 2018 15:18
woops
`    set idPatient, idSampleNormal, file(bamNormal), file(baiNormal), idSampleTumor, file(bamTumor), file(baiTumor), file("*.candidateSmallIndels.vcf.gz"), file("*.candidateSmallIndels.vcf.gz.tbi") into mantaToStrelka
`
results in bamfiles in the work dir and not the symlinks
should be
file("$bamTumor") indead of file(bamTumor)
wow i am bad at md
arontommi
@arontommi
May 03 2018 15:24
or maybe not
Hitesh Joshi
@evoxtorm
May 03 2018 15:27
Hey it's still showing "nextflow: command not found"
I've installed it properly why it is showing
Sendu Bala
@sb10
May 03 2018 15:28
After make compile && make pack && make install, the nextflow in that directory shows my changes in behaviour. But so does a version of nextflow installed with curl -fsSL get.nextflow.io | bash in a different directory. How is this possible? How can I test the behaviour of unmodified nextflow if I'm building my own?
Paolo Di Tommaso
@pditommaso
May 03 2018 16:13
simplest way to run for development is make compile and then ./luanch.sh
make test to run the unit tests