These are chat archives for nextflow-io/nextflow

20th
Aug 2018
Phil Ewels
@ewels
Aug 20 2018 10:19
What kind of thing do you want to know?
You're right, I can't find much in the docs either... :confused:
Karin Lagesen
@karinlag
Aug 20 2018 10:29
well...
I keep getting
WARN: Unknown directive threads for process run_fastqc
for things that I have defined in the config file as $task.threads
however, the value is being used, so it's not not working, but I am guessing that I should be doing something different
Alexander Peltzer
@apeltzer
Aug 20 2018 10:30
$task.cpus maybe?
Karin Lagesen
@karinlag
Aug 20 2018 10:31
correction: I use it as $task.threads inside the workflow script, but I have it defined as a process ...somethingorother? in the config file
I was under the impression that I could define whichever I wanted? Or am I confusing myself_
?
I wrote this part of the code last year, so I might be mixed up
Alexander Peltzer
@apeltzer
Aug 20 2018 10:32
I think you can define anything below params.XYZ
Karin Lagesen
@karinlag
Aug 20 2018 10:32
what do you mean with below?
Alexander Peltzer
@apeltzer
Aug 20 2018 10:32
where XYZ can be anything
so if you specify something like --XYZ 5you can access the parameter in your NXF script as $params.XYZ
Karin Lagesen
@karinlag
Aug 20 2018 10:33
ok, but these aren't params, these are process specific things I define in the profile specific config file
Alexander Peltzer
@apeltzer
Aug 20 2018 10:33
Yup - but IIRC you can't define new ones and have to use the directives available here: https://www.nextflow.io/docs/latest/process.html#directives
Karin Lagesen
@karinlag
Aug 20 2018 10:34
If so, I am confused why it works?
Alexander Peltzer
@apeltzer
Aug 20 2018 10:34
I am confused here as well ;-)
Karin Lagesen
@karinlag
Aug 20 2018 10:34
ok, to make things easier, here is the current code branch: https://github.com/karinlag/Bifrost/tree/2018-07-30-v0.2.0
so, I have a nextflow.config file that loads either standard og slurm config files depending on what I;m running
per process, I have options set in the config file, like cpus/threads/time/program names etc
and these are obviously actually used, because when I change the number of threads, in this case, it actually does use that number of threads
Alexander Peltzer
@apeltzer
Aug 20 2018 10:36
I see
But why do you need threads here? You could simply use the existing cpus directive and the warning is gone with no side effects ?
Karin Lagesen
@karinlag
Aug 20 2018 10:37
I see that I'm supposed to use withName instead of $ now
arontommi
@arontommi
Aug 20 2018 10:37
try removing treads
Alexander Peltzer
@apeltzer
Aug 20 2018 10:37
Yeah, but thats just a very recent change
Karin Lagesen
@karinlag
Aug 20 2018 10:37
I am likely to do that, but I was still wondering how this was actually put together
I can't find any docs on $task, which makes me a uncertain for how to proceed
Alexander Peltzer
@apeltzer
Aug 20 2018 10:39
You could even get rid of most of the cpus = 6 by setting that as default and changing it only in processes where there is need for that
Don't see the problem actually - your code works, just get rid of the threads (which doesn't exist), replace the $ with withName: and use the $task.cpus directive where applicable in your script - done :-)
Warnings should be gone then and you're fine :-)
I don't see tasks documentation either, some of the ones listed under directives here https://www.nextflow.io/docs/latest/process.html#directives are called via tasks.[directivename]
Karin Lagesen
@karinlag
Aug 20 2018 11:39
thanks for your comments, verifies and defines what I was sort of thinking :)
Alexander Peltzer
@apeltzer
Aug 20 2018 11:40
I guess we could help Paolo with documentation at some point too... he's already busy implementing for everyone :-)
Maxime Garcia
@MaxUlysse
Aug 20 2018 11:41
Yes, we should definitively help with the docs
And it's easy to add information
Karin Lagesen
@karinlag
Aug 20 2018 12:48
I could help out on that, but ... is this actually a nextflow invention, or does it stem from groovy,or somewhere else?
Sven F.
@sven1103
Aug 20 2018 12:52
it is part of the Nextflow DSL, so documentation regarding your matter i would expect in the Nextflow docs .
Karin Lagesen
@karinlag
Aug 20 2018 12:52
@sven1103 thanks, that helps quite a bit :)
Sven F.
@sven1103
Aug 20 2018 12:52
you are welcome :)
Karin Lagesen
@karinlag
Aug 20 2018 12:53
question: so, so far I take in params though a project specific config, and have compute specific things in the profile configs
so far I've used this to set up the command line for each process, so that people could specify themselves how to call trimmomatic, for instance
this has gone through $task set in the process
now that I get warnings basically saying "this is not how it's supposed to work" do any of you guys have suggestions for how to introduce this kind of flexibility?
Not sure I want to continue to have that flexibility, but thought I'd ask anyhow
Alexander Peltzer
@apeltzer
Aug 20 2018 12:56
You can do similar things, e.g. have a look at these configs here: https://github.com/nf-core/rnaseq/tree/master/conf
the base one defines generalized information for processes, whereas the hardware specific configurtion files e.g. binac define the hardware environment - these are put together as profiles on top of each other
You could also use containers for setting up things - that should not require users to specify where to find tools etc
Karin Lagesen
@karinlag
Aug 20 2018 13:01
I am thinking of going the conda route, seems to be accomplishing quite a bit of the same thing
but thanks for the link to the nf-core site :)
that cleared up quite a bit :)
Alexander Peltzer
@apeltzer
Aug 20 2018 13:03
Conda works fine too - its just not been there for a long time yet :-)
Good luck
Feel free to ask if there are some things coming up
Sven F.
@sven1103
Aug 20 2018 13:08

you can also group parameters very nicely in your configs, for example:

params {
    global_var1 = whatever
    fastqc {
         process_var1 = whatever
   }
  ...
}

and the call it with i.e. params.fastqc.process_var1 in your run-script

makes it cleaner to define cli params for the applications running inside a process
Karin Lagesen
@karinlag
Aug 20 2018 13:14
I had forgotten I could specify params elsewhere too :P
I guess that's what happens when you try to make v0.2 of a pipeline that you built about a year ago...
Karin Lagesen
@karinlag
Aug 20 2018 13:20
ok, question re conda:
The Conda environment feature is not supported by executors which use a remote object storage as a work directory eg. AWS Batch.
I use slurm
would that one use "remote object storage"....?
Kevin Sayers
@KevinSayers
Aug 20 2018 13:32
Nope it should work fine you just need to specify the cacheDir (https://www.nextflow.io/docs/latest/config.html#config-conda) on your slurm shared file system
Karin Lagesen
@karinlag
Aug 20 2018 13:34
thanks :)