These are chat archives for nextflow-io/nextflow

27th
Feb 2018
Trevor Tanner
@tantrev
Feb 27 2018 05:16
So I saw @pditommaso is speaking at this year's GCC BOSC 2018.
Congrats! Does that mean Nextflow by chance part of the Open Bioinformatics Foundation?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:27
@tantrev Hi there, thanks. No nextflow is not officially affiliated to BOF, but I've participated to several of their initiatives (conference, hackathon, google summer of code, etc)
Trevor Tanner
@tantrev
Feb 27 2018 07:43
Oh very cool. Speak of the devil actually, I was going to see what your thoughts might be on a Nextflow GSoC project. :smile:
I was thinking that NF could possibly benefit from a project focusing on making some more example pipelines with popular tools that haven't been used w/NF before (like those w/single-cell analysis, transposons, etc. in bioinformatics). The computational chemistry community could especially also really benefit from NF. Maybe also systematically create examples of NF's cooler features?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:45
we participated a couple of times, this year it's too late .. :)
Trevor Tanner
@tantrev
Feb 27 2018 07:47
Ah shucks! Oh well, glad I asked. :) I'm always telling people how great NF is haha.
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:47
that's good !
what's your org ?
Trevor Tanner
@tantrev
Feb 27 2018 07:49
I went to school at the University of Utah, but am just taking some classes at the local community college (SLCC) at the moment.
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:49
Cool
planning to become a bioinformaticians, I guess
Trevor Tanner
@tantrev
Feb 27 2018 07:52
Systems bio is looking like the best fit at the moment. Wanna learn some experimental skills too tho, so maybe synthetic biology? It all seems the same after a while lol
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:53
LOL
ok, great to meet you at bosc 2018 if you will join
Trevor Tanner
@tantrev
Feb 27 2018 07:54
Yeah, I hope to as well! It'd be great to see your talk. We'll see if I can find a GSoC project that's a good fit lol.
Paolo Di Tommaso
@pditommaso
Feb 27 2018 07:54
:+1:
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:31
hello, quick question. When Nextflow gives an error saying that a process is missing some output files, this check I guess is performed after the job is finished, is that correct ?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:32
quick answer: yes
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:33
I am experiencing some weird situation where I am running thousands of jobs on AWS Batch and for some reason it fails saying that for a task is missing an output file. The problem is that the task terminated correctly and on the S3 path there are all the output files and the .command.log and .exitcode (which is zero btw)
given that the workflow is running thousands of jobs and retrieving also some of these from the cache, could it be some sort of timeout thing of the API requests on S3 and NF does not have an answer back in time ?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:35
mmm, do you mean it says there are missing files, but they are stored in s3
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:36
yes the task completed properly, all files are there (even the one NF is saying is not) as well as the control files from NF
everything is in good order
no errors in the log, exit code 0
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:37
hence if you restart it, it should continue
does it ?
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:38
no he re-runs everything even with the -resume
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:38
holy crap, why ?
any weird character in the file path ?
no he re-runs everything even with the -resume
even jobs that returned no error ?
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:43
yes
not at all, it’s the usual path on S3 we use as temporary dir for our pipelines
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:45
mmmm, something wrong is happening
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:46
I am using the GitHub master version at this commit nextflow-io/nextflow@a0dbc75
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:47
is it only a problem only related to the current master ?
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:47
now I am re-running only a part of the jobs to see if I still get the same thing
I can check with the latest stable release
when this run is finished
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:48
as you are saying it seems at system problem, are you able to replicate with a test case ?
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:51
re-running now, let me see if with less jobs submitted it will still raise a similar error
Paolo Di Tommaso
@pditommaso
Feb 27 2018 09:52
ok, I'll wait a more stable problem assessment
Francesco Strozzi
@fstrozzi
Feb 27 2018 09:53
yes it’s happening again
reverting back to the last stable version, cleaning the cache and starting with a new fresh working environment to see if it’s still there
Francesco Strozzi
@fstrozzi
Feb 27 2018 10:35
ok there was something weird with the local cache, now is running and retrieving correctly from the cache as well. So it was a local problem, not S3 or NF related.
Paolo Di Tommaso
@pditommaso
Feb 27 2018 10:35
:tada:
NF bug free day ! :joy:
Francesco Strozzi
@fstrozzi
Feb 27 2018 10:35
as always ;)
Phil Ewels
@ewels
Feb 27 2018 12:25
@pditommaso - did you add the Nextflow logo to the Atom file-icons package?
image.png
just noticed this ^ - awesome!
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:28
Just noticed this at the moment too
You beat me to it
Francesco Strozzi
@fstrozzi
Feb 27 2018 12:34
:+1:
that’s cool
Toni Hermoso Pulido
@toniher
Feb 27 2018 12:36
Hello, one question, when a process is submitted to a cluster system such as SGE, is Nextflow storing somewhere the cluster queue ID assumed by that process ?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:43
@ewels you are a star !
@toniher the job id do you mean ?
Phil Ewels
@ewels
Feb 27 2018 12:43
@pditommaso - it wasn't me! I didn't make them, I just noticed it appearing in my editor..
Toni Hermoso Pulido
@toniher
Feb 27 2018 12:43
@pditommaso yep!
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:44
@ewels I didn't do!
@toniher yes, but it's reported only in the trace file
Toni Hermoso Pulido
@toniher
Feb 27 2018 12:45
@pditommaso ah, good to know. Thanks!
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:46
@pditommaso who did it then?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:46
no idea!
Phil Ewels
@ewels
Feb 27 2018 12:46
Looks like it was the main author after it was added to GitHub linguist: file-icons/atom@3638b98
or originally file-icons/source@4047532
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:48
:+1:
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:48
awesome
how to install that icons?
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:50
it's automatic
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:50
I have none of them
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:50
you need to have the file icons package in atom up to date
Phil Ewels
@ewels
Feb 27 2018 12:55
yes, install the file-icons package in atom
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:55
:tada: :tada: :tada:
Maxime Garcia
@MaxUlysse
Feb 27 2018 12:55
@pditommaso It's this one
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:55
it works !
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:55
tx
now it's only missing on Vscode
Tintest
@Tintest
Feb 27 2018 12:58
Does nextflow support the -resume option if the main nextflow call some slaves nextflow ?
I specify the -resume option in my main nextflow, and in the command calling for the slaves, but it does not seem to work
Paolo Di Tommaso
@pditommaso
Feb 27 2018 12:59
what do you mean slave nextflow ?
Tintest
@Tintest
Feb 27 2018 13:00
I call a nextflow script where there is other nextflow scripts called
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:01
I see, that's not a built-in feature therefore NF cannot infer automatically if a job was launched with or without resume
Tintest
@Tintest
Feb 27 2018 13:02
So, you will advise me to merge everything into only one script, if I want to be able to use the resume option ?
Bioninbo
@Bioninbo
Feb 27 2018 13:03
Hello. I have a multiQC process that seems to not be cached well. When I modify a downstream process the multiQC process is often re-submitted. Any idea why and how to avoid it?
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:07
could it be this nextflow-io/nextflow#610
Bioninbo
@Bioninbo
Feb 27 2018 13:07
I have similar caching issues with a process using intersectBed (bedtool)
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:07
have you checked the above issue ?
Bioninbo
@Bioninbo
Feb 27 2018 13:08
in the multiqc process I used .flatten().toList()
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:09
well, hard to say without a replicable test case
Bioninbo
@Bioninbo
Feb 27 2018 13:09
I used this input: file ('fastqc/*') from fastQC_reports_before_trimming_for_multiQC.flatten().toList() file ('fastqc/*') from fastQC_reports_after_trimming_for_multiQC.flatten().toList()
so yes I used the somDir/* pattern
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:10
what version are you using ?
Bioninbo
@Bioninbo
Feb 27 2018 13:10
version 0.27.0 build 4751
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:11
then nextflow -self-update and try it again
Bioninbo
@Bioninbo
Feb 27 2018 13:11
ok. Thanks!
Bioninbo
@Bioninbo
Feb 27 2018 13:32
This caching issue seemed fixed now thanks. However after updating via nextflow -self-update I now get this error message: /usr/bin/nextflow: line 429: /home/usr/.nextflow/tmp/launcher/nextflow-one_0.27.6/usr-VirtualBox/classpath-8db92d5d54a9898ec8b106741735b1b5: Permission denied The classpath file do not exists. It does not prevent the program from running though.
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:35
weird
try to delete them tmp path
rm -rf /home/usr/.nextflow/tmp
and run NF again
Bioninbo
@Bioninbo
Feb 27 2018 13:37
yes it worked, no warning messages now. Thanks @pditommaso !
Paolo Di Tommaso
@pditommaso
Feb 27 2018 13:37
:+1:
kevbrick
@kevbrick
Feb 27 2018 14:58
A quick question. I have a process with separate pass/fail output files. Sometimes, there may be no fails, so the "fail" output will be either absent or empty. How can I get nextflow to accept this without an error causing the process to fail ?? Many thanks ...
Félix C. Morency
@fmorency
Feb 27 2018 15:38
@kevbrick With something like
output:
file "failing.txt" optional true
who is using NF in production, please give a try to this RC
Félix C. Morency
@fmorency
Feb 27 2018 15:54
\o/ Thanks!
Paolo Di Tommaso
@pditommaso
Feb 27 2018 15:55
:)
kevbrick
@kevbrick
Feb 27 2018 15:59
Thanks @fmorency !!
Félix C. Morency
@fmorency
Feb 27 2018 16:08
:+1: