These are chat archives for nextflow-io/nextflow

31st
Oct 2018
Maxime HEBRARD
@mhebrard
Oct 31 2018 06:52
Hello
Small issue: one process create a folder containing multiple files... I wish to publish some of the files in another directory, how can I check or create my directories prior of publishDir ?
Paolo Di Tommaso
@pditommaso
Oct 31 2018 07:18
depending the file name ?
Maxime HEBRARD
@mhebrard
Oct 31 2018 07:34
hmm... ok I will test that
Maxime HEBRARD
@mhebrard
Oct 31 2018 07:51
sounds exactly what I wanted :)
Paolo Di Tommaso
@pditommaso
Oct 31 2018 07:51
nice
Maxime HEBRARD
@mhebrard
Oct 31 2018 08:00
is there a command to remove some files from work directories after the process end ?
Paolo Di Tommaso
@pditommaso
Oct 31 2018 08:00
under dev
Maxime HEBRARD
@mhebrard
Oct 31 2018 08:01
nice
Maxime HEBRARD
@mhebrard
Oct 31 2018 08:19
can I check if a file exists as condition for my process ?
when:
      !file("${params.outdir}/*.txt").exists()
Paolo Di Tommaso
@pditommaso
Oct 31 2018 08:21
not sure, try it
in any case there should not .exists()
because if the file exists the glob is resolved and returns a list of files => true
Maxime HEBRARD
@mhebrard
Oct 31 2018 08:23
oh ... yes I try it and get "No signature of method: java.util.LinkedList.exists()"
the script is running with when: !file("...) ... I will need more test to see if the condition actually works ^^"
Paolo Di Tommaso
@pditommaso
Oct 31 2018 08:26
:+1:
Karin Lagesen
@karinlag
Oct 31 2018 10:22
So, I am ending up in a situation here where things quite quickly use up massive amounts of space
I am thinking of going over to using link mode for publishDir
I am used to thinking of hardlinks as a "bad thing", thus anybody got suggestions for why this could be bad in this case?
Riccardo Giannico
@giannicorik_twitter
Oct 31 2018 10:38
@karinlag why not symlink ?
Rad Suchecki
@rsuchecki
Oct 31 2018 10:47
hardlinks are great, just don't travel across file systems. if they work in your context - fine, but porting the pipeline elsewhere may not work without changing that to symlink/copy
Winni Kretzschmar
@winni2k
Oct 31 2018 12:08

if you want to test your task commands, save them as separate bash scripts, and then test individually

cheers!

@karinlag why not go ahead and use "copy" mode instead (if you have the space)? That works across file system boundaries...
Karin Lagesen
@karinlag
Oct 31 2018 14:41
@winni2k My problem is that I have a project where the start files are already close to 700M
with the pipeline I have, that just about quintuples during processing
so, if I publish all that, I end up with 7TB just for that project
however, I don't want to just copy out results, I want people to have a chance to check what happens during processing
copy out the end results, that is
Chadi Saad
@chadisaad
Oct 31 2018 16:01
Hello, does any one have this error ? "container creation failed : mount error too loop device file exists"
sometimes i got it randomly
I re-execute the same nextflow, with same parameters, error disappear !