These are chat archives for nextflow-io/nextflow

27th
Jul 2016
Sandeep Shantharam
@machbio
Jul 27 2016 00:13
Is it possible to post the parameters required for a pipeline through json or yaml file ?
Paolo Di Tommaso
@pditommaso
Jul 27 2016 08:31
@machbio json or yaml files no, but if the syntax does not matter you can use the nextflow config file for that (that's very similar to json)
For example:
params {
  alpha = 1 
  beta = 'Hello world' 
}
Can this be an alternative? could you elaborate your use case ?
Mokok
@Mokok
Jul 27 2016 11:57
@pditommaso is there a way to manually interrupt a workflow execution (durring the execution, or if it has be scheduled, during the "waitting" time) ?
Sandeep Shantharam
@machbio
Jul 27 2016 12:10
@pditommaso looking for dynamic execution, like having a single rna seq pipeline nextflow file - but running multiple projects containing different samples each time.. To be able to post json or yaml file containing details of samples directly into nextflow like "nextflow - - samples /path/to/jsonfile" .. I am on gitter mobile, I can mock up and create an issue, if you think it's viable case..
Continuing on that i can see that nextflow can accept file in parameters, which I will have to parse with the first process using python.. But a direct parsing with nextflow helps dynamically running the pipelines..
Paolo Di Tommaso
@pditommaso
Jul 27 2016 12:25
@Mokok You can simply kill it by using CTRL+C, the second part of your questions is not clear
Paolo Di Tommaso
@pditommaso
Jul 27 2016 12:32
@machbio currently you can map your json to nextflow params with this one liners
new groovy.json.JsonSlurper().parseText('{ "alpha": 1, "beta": "Hello"}').each { k, v -> params[k] = v }

println params.alpha
println params.beta
however still not clear what's the benefits of json/yml over a nextflow config params format
Sandeep Shantharam
@machbio
Jul 27 2016 12:38
@pditommaso I will try out the above today.. The advantages is, if you are building a server side executioner (with nextflow) with a client app to be used would post json object that my server backend can directly post to nextflow and not worrying about creating configure for each run.. I am on mobile, I will create an issue and you can evaluate much clearly..
Paolo Di Tommaso
@pditommaso
Jul 27 2016 12:38
ok, thanks
Mokok
@Mokok
Jul 27 2016 13:30

The final purpose is to integrate nextflow as a data-processing scheduler module. So, i'm checking how needs are covered.
For example, there are some extra-needs like
{1} being able to specify the date when the job has to start
{2} how to interrupt it during the running phase or during the waiting phase ( when the job haven't started yet)
{3} generating execution report (by job and by inner task : execution status and some more information)
{4} if a task can interrupt a parent job or an other one (with an given ID ?)
{5} more generally, if a task can know about other task that as been submitted (completion, exec status, ...) and then it the task can have some 'power' on it (kill/cancel, restart, interrupt+modify+resume,...)

sorry for the long post (and big questions!)

(ofc theses needs could be added, i just wonder if they are already in, or if there is way easier than an other to do one of these...like 'cron' for the start date for example)
/end
Paolo Di Tommaso
@pditommaso
Jul 27 2016 13:32
{1} nextflow is not a chron schedule, so you will need to implement
{2}you can interrupt at any time without losing partial computation
{3} here
{4} no
{5} each tasks is isolated by each other by design, so no
Mokok
@Mokok
Jul 27 2016 13:38
ok ! thanks for the good answers (again)
is there no way to work with the pid associated with the task (or the task_id/native_id that i see in the execution report you just linked) ?
Paolo Di Tommaso
@pditommaso
Jul 27 2016 13:41
No, out-of-the-box. A Nextflow pipeline is based on a functional paradigm thus you are not supposed to have such kind of control typical of a imperative approach
Mokok
@Mokok
Jul 27 2016 13:43
ok thanks for the accurate answer(s)
Paolo Di Tommaso
@pditommaso
Jul 27 2016 13:44
welcome !
Mokok
@Mokok
Jul 27 2016 15:36
in saw that depending on the resource requirement specification of a job its priority may change (the bigger the constraint is, the lower is the priority). But i didn't find any way to SET this priority, is there one ?
Paolo Di Tommaso
@pditommaso
Jul 27 2016 15:43
job priority is delegated to the underlying scheduler e.g. PBS, nextflow manages the tasks synchronisation but does not the scheduling policy
Mokok
@Mokok
Jul 27 2016 15:47
ok, using clusterOption and qsub's -p <-1024 to +1023> parameter then.
Paolo Di Tommaso
@pditommaso
Jul 27 2016 15:50
that's an option
Mokok
@Mokok
Jul 27 2016 15:51
ofc, it was in my particular case ^^
Jason Byars
@jbyars
Jul 27 2016 22:47
If messages are written to stderr, is that sufficient for a process to fail, or does some operation need to have a non-zero exit code?