thanks @mes5k that's very helpful, I'm trying this approach, will let you know how I make out
Good morning. Is it possible to set nextflows
work dir to be somewhere else? I'm processing a long timeseries and I'm afraid I cannot fit it entirely on my disk
of course
-w cli option
Ah, I see. And from
nextflow.config?
Future me will be very happy to have everything store in a config file :grimacing:
nevermind, I found it out ;)
workDir = '/some/path'
:+1:
Is there anything nextflow can't do? :clap:
coffee :)
We managed to get it for free :confetti_ball:
But I'll be happy if it would cook dinner for me
LOL
Actually it could be done with some lab automation system?
controlled by nextflow
Another question: is nextflow handling of glob patterns different from the way bash treats them?
Because when using certain patterns with
fromFilePairs nextflow does not return any file, but if is use the same pattern on my shell, I can find them
NF relies on Java glob pattern, which may not be identical to BASH
Well that explains a lot then. I guess exclude patterns are not supported by java
Or rather they work differently
I presume I cannot use a regex instead?
yes, you can
specify a pattern object as
~/your-regex/ (if I'm not wrong) to fromPath
That's excellent :+1:
@pditommaso Said and done! Here's the pipeline: https://github.com/oskarvid/nextflow-GermlineVarCall/blob/master/bwamem.nf
fantastic
so what's the problematic part ?
When MergeBamFiles takes its input files it seems to take two random files, but it needs to take the correct pairs.
let me see
so there's eight pairs, one pair per lane, one file is unmapped and the other is mapped
do you mean
MergeBamAlignment ?
+yeah
basically
BwaMem_output and FastqToSam_output, right ?
yes
the easiest way to handle this is to keep the
pair_id along with the output file
i.e don't use pair_id and pair_id2?
yes, but you need to specify in the output, for example
in the
Bwa_mem process, replace
output:
file "bwamem.sam" into BwaMem_output
with
output:
set pair_id, file("bwamem.sam") into BwaMem_output
the same for the
FastqToSam process
then, since the process execution is parallel, the output ordered
hence you will need to create a channel containing the sam and bam for the same
pair_id
does make sense ?
to begin with, it doesn't like that I'm using the channel "reads" for both bwa and fastqtosam
that's why I made two earlier, but I don't need to? or shouldn't?
do you mean this
?
yes
you can have the same channel as input for more than a process, but you can simplify that as shown below
Channel
.fromFilePairs( "/data/workspace/data/Samples/NA12878-rep7_S7_L00*_R{1,2}_001.fastq.gz", flat: true)
.into { reads; reads2 }
It's still not clear to me what I need to do, but I
oops
but I'm going home now for today, will look at it tomorrow
:ok_hand:
@pditommaso do you have an ETA for 0.25.3? :D
Hi! I started to using nextflow which I find amazing! I wanted to create a small cluster for aws just for testing using the following CMD: nextflow cloud create test-cluster (up to 3 t2.micro instances). I'm having problems to run the CMD as it complains saying: "ERROR ~ Cannot cast object 'null' with class 'null' to class 'int'. Try 'java.lang.Integer' instead". Any ideas how to solve this? Thanks in advance! :smile:
@fmorency hopefully, thu or fri
@pditommaso awesome thanks
@dsoronellas yes, we just spotted this issue nextflow-io/nextflow#408
use the command in the last comment
ok! I go for it! Thanks for the immediate response hehe
Is it possible to clean only the older results from nextflows cache?
yes, see
nextflow clean
yes, but does it clean only the older ones?
I'm afraid to type it :fearful:
check the help
I see
;)
You can execute a dry run to see what will be removed.
It will list the files that will be removed without removing them.
fantastic
Hi,
I managed to reproduce my storeDir issue by an unfortunate coincidence... This time, the whole pipeline was terminated due to critical failure in some process. Because of that, some upstream processes that were copying results to storeDir were also terminated/aborted etc. I relaunched the pipeline (actually without -resume flag) and it took off with whatever was in the storeDir folder - regrardless of whether the upstream was aborted, results were incomplete or whatever. It seems that the only thing nextflow cares in such cases is the content of storeDir. Is it expected behavior? Is there a way to make nextflow check if copying to storeDir went ok?
unfortunately this is a side effect of
storeDir
do you have more than a process writing to the same
storeDir ?
Yes
I mean multiple instance of the same process
you can try to mitigate this problem with
errorStrategy = 'finish'
Let's say, when we are mapping chunks ,
bam files go to the same storeDir
however, this is the reason why I suggest to use
publishDir in place of storeDir whenever possible
Can I do
errorStrategy = { task.attempt>2 ? 'finish' : 'retry' } ?
yes
Ok, I'll try to read about
publishDir behavior, and see if we could rewrite the pipeline in accordance with the best practices ...
you should use
storeDir only when it's needed a cache across different execution of the pipeline
Thank you!
:+1:
what do you mean like a cache for different executions?
say, if i'd want to change some parameters and relaunch?
something like that>?
for example you download a big database file, and you want to keep in the local storage
if i'd want to change some parameters and relaunch?
no in this case you won't need
storeDir
just use the
-resume mechanism
ok, I need to read more about
publishDir to see the difference
do
-resume check the contents of the work directory?
if there is no
storeDir?
yes
oh! I see - i didn't know that
that's guaranteed to be consistent
and does
-resume check things like .exitcode ?
like, if the process was actually successful ?
or is it just content-based?
check the task exitcode and the existance of the expected files
that sounds exactly what we'd need! I have no idea why they used so many
storeDir in the pipeline to begin with ...
nice
I didn't write the pipeline from scratch
I'm just trying to run and adjust it on a cluster
which is the pipeline ?
distiller
I would keep storeDir only here
Now I see, what kind of cache you've meant ...
Thank you so much Paolo!
you are welcome
When I run this:
./nextflow cloud create optitype-cluster -c 4
I get the following error:
> cluster name: optitype-cluster
> instances count: 4
> Launch configuration:
- driver: 'aws'
- imageId: 'ami-cc2d64da'
- instanceType: 'c4.xlarge'
- keyName: 'xxxxxxxxx'
- securityGroup: 'xxxxxxxxx'
- subnetId: 'xxxxxxxxx'
- userName: 'ubuntu'
Please confirm you really want to launch the cluster with above configuration [y/n] y
Fetching EC2 prices (it can take a few seconds depending your internet connection) ..
ERROR ~ Cannot cast object 'null' with class 'null' to class 'int'. Try 'java.lang.Integer' insteadAny thoughts on why this might be happening? This was working fine for me a couple weeks ago. Any help would be much appreciated!
it seems AWS pushed some dirty data in the price file
nextflow-io/nextflow#408
there's a temporary workaround in the last comment
@pditommaso Doing the following worked.
export NXF_VER=0.25.3-SNAPSHOTThanks!
Running this command on the master node
./nextflow run nmdp-bioinformatics/flow-Optitype \
--with-docker nmdpbioinformatics/flow-OptiType \
--outfile hli-optitype.csv \
--bamdir s3://bucket/s3/data \
--datatype dnaReturns the following errors:
N E X T F L O W ~ version 0.25.3-SNAPSHOT
Pulling nmdp-bioinformatics/flow-Optitype ...
downloaded from https://github.com/nmdp-bioinformatics/flow-OptiType.git
Launching `nmdp-bioinformatics/flow-Optitype` [sleepy_jang] - revision: 6fcb330fe1 [master]
---------------------------------------------------------------
NEXTFLOW OPTITYPE
---------------------------------------------------------------
Input BAM folder (--bamdir) : s3://bucket/s3/data
Sequence data type (--datatype) : dna
Output file name (--outfile) : hli-optitype.csv
[warm up] executor > ignite
ERROR ~ ip-xxxxx-xx: ip-1xxxx-xx-xx: Name or service not known
-- Check script 'main.nf' at line: 68 or see '.nextflow.log' file for more detailsWhat's the best way to share log data? Thanks, Mike.
open an issue on GH and upload there the log file, thanks
nextflow-io/nextflow#409