These are chat archives for nextflow-io/nextflow

31st
Jan 2017
Trevor Tanner
@tantrev
Jan 31 2017 06:47
Is there any way to pass NF run-time arguments like "-resume" in the NF console?
Paolo Di Tommaso
@pditommaso
Jan 31 2017 08:43
@tantrev nope
Trevor Tanner
@tantrev
Jan 31 2017 17:16
That would be a really nice feature :smile:
Paolo Di Tommaso
@pditommaso
Jan 31 2017 17:17
Feel free to open feature request on GH !
Trevor Tanner
@tantrev
Jan 31 2017 17:21
Done. :+1:
Paolo Di Tommaso
@pditommaso
Jan 31 2017 17:21
Great!
Are u not fluent with Java/Groovy ?
:)
Trevor Tanner
@tantrev
Jan 31 2017 17:23
Haha I'm pretty rusty. I mainly just have processes in my workflow that take a while to compute.
Executing NF from bash works, the NF console just executes so much faster. :smile:
Paolo Di Tommaso
@pditommaso
Jan 31 2017 17:24
Yes, that's true.
though, there could be weird side effects when re-running scripts over time
I would suggest only for learning/training purposes
Trevor Tanner
@tantrev
Jan 31 2017 20:26
I had a feeling you might disapprove lol. Is there an easy way to specify something like the resume boolean directly in the NF script?
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:27
not really, but it isn't supercritical
I can give a try to it
Trevor Tanner
@tantrev
Jan 31 2017 20:29
no pressure :smile:
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:30
ideally putting resume = true in the nextflow.config should work
Trevor Tanner
@tantrev
Jan 31 2017 20:30
ohh, very nice - I'll give that a try
Trevor Tanner
@tantrev
Jan 31 2017 20:37
unfortunately, that seems to cause an error: http://pastebin.com/raw/RPCD9iqX
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:37
fu**
Trevor Tanner
@tantrev
Jan 31 2017 20:38
yeah
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:38
add that stack trace in the issue you have opened please
Trevor Tanner
@tantrev
Jan 31 2017 20:38
will do
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:38
thanks
amacbride
@amacbride
Jan 31 2017 20:58
So what I've discovered is that I consistently see the failure I mentioned (can't find the output file) when using multiple cluster nodes and the scratch directive set to a node-local file (including 'ram-disk') -- so far, it seems to be working when using an NFS-mounted volume as the scratch dir.
BTW, I'm no longer seeing the 'nxf-kill' error when using 0.23.2-SNAPSHOT
Paolo Di Tommaso
@pditommaso
Jan 31 2017 20:59
um
but does not solve the file not found failure
amacbride
@amacbride
Jan 31 2017 21:02
Well, it works around it (though it's not optimal for me to use NFS for the scratch dir, given the file sizes involved). I'll see if I can come up with a nice small test case and file an issue on it.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:03
it's always the same process reporting the problem
or they change?
amacbride
@amacbride
Jan 31 2017 21:05
I did just discover a new issue with 'set' and 'val' with 'stdout' -- 'set' gives me a warning, but works, while 'val' fails:
(set)
WARN: Output `set` must define at least two component -- Check process `getReadInfo`

-----


(val)
WARN: Missing value declared as output parameter: stdout -- Execution is retried (1)
[34/501f5e] Re-submitted process > getReadInfo (E11_VF39080-A)
ERROR ~ Error executing process > 'getReadInfo (E11_VF39080-A)'

Caused by:
  Missing value declared as output parameter: stdout
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:06
in there's only on component don't use set
amacbride
@amacbride
Jan 31 2017 21:06
@pditommaso In this case, it's always the same process, since I pared things down. I've seen it in the past on various different processes.
Ah, I see my mistake -- I was using 'val stdout' instead of just 'stdout'
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:09
ok
amacbride
@amacbride
Jan 31 2017 21:11
ok, I guess I don't understand the declaration syntax. I'm not getting an error from:
 output:
                set sample_name, file(bam) into readinfo_samplename1, readinfo_samplename2
                stdout into readinfo_downsample_percentage1, readinfo_downsample_percentage2
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:12
it looks fine, what's the problem?
amacbride
@amacbride
Jan 31 2017 21:12
How should the second output channel be declared? I want to send stdout into two channels.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:13
stdout into readinfo_downsample_percentage1, readinfo_downsample_percentage2 is not working?
amacbride
@amacbride
Jan 31 2017 21:13
N E X T F L O W  ~  version 0.23.2-SNAPSHOT
Launching `../clia-pipeline/nextflow/modular.nf` [desperate_galileo] - revision: a470acfa2a
ERROR ~ expecting '}', found ',' @ line 436, column 46.
   eadinfo_downsample_percentage1, readinfo
                                 ^

1 error
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:15
um
try this
stdout() into (readinfo_downsample_percentage1, readinfo_downsample_percentage2)
amacbride
@amacbride
Jan 31 2017 21:20
That seems to be working, thanks.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:20
unfortunately that's a glitch with that syntax nextflow-io/nextflow#141
amacbride
@amacbride
Jan 31 2017 21:22
Gotcha. Might be worth putting a note in the documentation.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 21:22
good idea
amacbride
@amacbride
Jan 31 2017 22:18

Today is my warnings cleanup day, so on another note, I was looking at TaskProcessor.groovy, lines 763-766:

if( invalid ) {
            log.warn "[$task.name] Store dir can be used when using 'file' outputs"
            return false
        }

...is that supposed to be, "can only be used when using 'file' outputs" ?

Paolo Di Tommaso
@pditommaso
Jan 31 2017 22:19
oops
well done
amacbride
@amacbride
Jan 31 2017 22:19
:)
Paolo Di Tommaso
@pditommaso
Jan 31 2017 22:19
thanks!
fixed
amacbride
@amacbride
Jan 31 2017 22:20
you rock
Paolo Di Tommaso
@pditommaso
Jan 31 2017 22:20
;)
amacbride
@amacbride
Jan 31 2017 23:03
Ah, sadly, I'm still seeing the problem when using an NFS scratch dir, but it's more infrequent. In answer to your question above, it's happening at multiple steps of the pipeline, not just one. And it varies from run to run.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:04
what cluster are u using ?
amacbride
@amacbride
Jan 31 2017 23:05
slurm
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:05
with Docker ?
amacbride
@amacbride
Jan 31 2017 23:05
yup
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:05
really weird
let me check one thing
amacbride
@amacbride
Jan 31 2017 23:06
I've been looking into the implementation of collectOutFiles, but hadn't seen anything useful yet.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:07
I'm still thinking that's related to NFS latency
I've have seen very eager caching on NFS
up to minutes under an heavy load
anyhow you may want to try process.cleanup = false in your config file
amacbride
@amacbride
Jan 31 2017 23:09
what does that do?
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:09
disables the scratch cleanup
just to be sure there isn't something wrong with docker/root permission
amacbride
@amacbride
Jan 31 2017 23:12
That's always a possibility, I will pursue that avenue as well -- I may try mounting it with noac and other caching options disabled.
Cool, I will try disabling cleanup as well.
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:13
ok, let me know how it goes
amacbride
@amacbride
Jan 31 2017 23:23
process {
    cleanup = false
}
...gave me a bunch of errors. How would I use the process.cleanup facility in my nextflow config file?
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:24
how really ?
amacbride
@amacbride
Jan 31 2017 23:25

The full file reads:

process {
        executor = "slurm"
        queue = "duo"
        //scratch = "/nfs/zfs_IT/nextflow/tmp"
        scratch = true
        cleanup = false
}

WARN: Unknown directive cleanup for process startup

(and a similar warning for each process in the script)
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:26
are u using the latest version right ?
amacbride
@amacbride
Jan 31 2017 23:26
N E X T F L O W  ~  version 0.23.2-SNAPSHOT
yup
Paolo Di Tommaso
@pditommaso
Jan 31 2017 23:27
um, you are right
ok, though the warning message the directive is applied
you can try it
then, I will remove the warning