These are chat archives for nextflow-io/nextflow

19th
Oct 2018
Luigi Grassi
@luigra
Oct 19 2018 12:47
Hello I am trying to run my nextflow pipeline on hpc, using sge as executor; it worksfine with the option process.stageInMode = 'copy' declared in the config file but w/o it is not able to locate any file index,fastq etc.. Is there any other configuration parameters I should set to avoid process.stageInMode = 'copy' ? (Coping all the needed file for each job is quite time consuming) Thanks in advance
Paolo Di Tommaso
@pditommaso
Oct 19 2018 12:49
try without process.stageInMode and use instead process.scratch = true
Luigi Grassi
@luigra
Oct 19 2018 12:53
Thanks Paolo, I have the same error: it does not link any file in the working directory
Paolo Di Tommaso
@pditommaso
Oct 19 2018 12:54
well if so use process.stageInMode = 'copy' :smile:
Luigi Grassi
@luigra
Oct 19 2018 12:57
:-( Ok Thanks a lot and congratulations for nextflow it is amazing and oversimplifies my work
Paolo Di Tommaso
@pditommaso
Oct 19 2018 12:57
thanks !
btw using stageInMode solves your problem or it doesn't ?
Luigi Grassi
@luigra
Oct 19 2018 13:01
it does but it make the process slower. It is copying for each process all the files so for an alignment it copies all the fastq files and all the index files
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:02
how usually do you run jobs in your cluster?
above all why simlinks are not allowed ?
Luigi Grassi
@luigra
Oct 19 2018 13:03
I don't know I 'll check by creating a symlink from one node
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:03
:+1:
Luigi Grassi
@luigra
Oct 19 2018 13:04
I am just migrating the pipeline I used on my workstation to the hpc
Luigi Grassi
@luigra
Oct 19 2018 13:36
Paolo I am able to create simlink in the nodes running the pipeline and I can read the partition where the files are but if I define them as input w/o process.stageInMode = 'copy' the pipeline does not start, on contrary if I copy these file on my home the pipeline works. Is there any filesystem issue or setting that I have to specify? Thanks again
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:39
let me understand, does your institute cluster has a shared file system accessible from all the computing nodes?
Luigi Grassi
@luigra
Oct 19 2018 13:39
yes it has
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:40
are you launching the NF pipeline in a directory in such file system ?
Luigi Grassi
@luigra
Oct 19 2018 13:41
my directory is currently in my home
should I move it in the same location of the files?
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:42
is your home directory accessible from all cluster node via the shared file system ?
Luigi Grassi
@luigra
Oct 19 2018 13:43
It should be
but I am not 100% sure
I'll double check
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:43
ok, find the correct answer :)
if no --> you have to launch NF in a directory accessible by the cluster nodes.
Luigi Grassi
@luigra
Oct 19 2018 13:44
ok
thanks a lot
Paolo Di Tommaso
@pditommaso
Oct 19 2018 13:44
:ok_hand:
Luca Cozzuto
@lucacozzuto
Oct 19 2018 15:32
hello! I have a question about ignoring errors with Nextflow
when this happens the steps that require the channel from the failed processes are not executed (Even the ones that mix more channels). Is there a way to output an empty channel so that the can go through?
micans
@micans
Oct 19 2018 16:44

I'm pondering something similar to @lucacozzuto .. I have a process that may transiently fail; at the moment the script checks failure and if it fails it generates a file lostcause.txtthat is sent to a process that collects all lost causes. To do forwarding I think the script needs to have zero exit status, but if the script ends with zero exit status NF will not retry it on resume. I have a faint suspicion that I am being unreasonable. [I'm using 'ignore' rather than 'retry' at the moment]. In you case Luca, can't you do this at the script level? I do this:

        if bash -euo pipefail irods.sh -t ${params.studyid} -s ${samplename}; then
          true
        else
          stat=\$?
          tag='UNKNOWN'
          if [[ \$stat == 64 ]]; then tag='NOFILES'; fi
          echo -e "${samplename}\\tirods\\t\$tag" > ${samplename}.lostcause.txt
        fi

I have some hope that @pditommaso will now tell me a much better way to organise this :grinning:

I combine the above with this output section:
        output:
            set val(samplename), file('*.cram') optional true into ch_cram_files
            file('*.lostcause.txt') optional true into ch_lostcause
micans
@micans
Oct 19 2018 17:09
mmm, I guess I don't need files to begin with, can just make them values.
Chris Fields
@cjfields
Oct 19 2018 20:09
Can't expect syntax to always agree between programming languages or we'd only have one :wink:
Oh, don't I know it :)
BTW, thx @pditommaso for the issue update