These are chat archives for nextflow-io/nextflow

8th
Mar 2018
Phil Ewels
@ewels
Mar 08 2018 06:59
Good morning all :) Regarding nextflow config, I have two questions / feature requests ;)
1) Is it possible to also print variables which are defined in the nextflow script, in addition to the nextflow.config files?
2) Is it possible to output the data in a format that is easier to parse? eg. YAML or JSON or something..
Shellfishgene
@Shellfishgene
Mar 08 2018 08:31
I'm still stuck on above error, I have no idea why that's appearing. The line I quoted above should be fine, no?
Paolo Di Tommaso
@pditommaso
Mar 08 2018 08:49
@Shellfishgene it looks like something wrong in the command script not the input declaration
if I'm not wrong a cause of that could be a bash snippet such as
x=$(do_something_here)
Shellfishgene
@Shellfishgene
Mar 08 2018 08:51
@pditommaso Yes, that was it. I had $(id) in the script.
Paolo Di Tommaso
@pditommaso
Mar 08 2018 08:51
hence fixed ?
@ewels good morning !
1) not possible, the script execution depends on the config object, therefore it must be parsed before the script itself
Shellfishgene
@Shellfishgene
Mar 08 2018 08:54
Yes, thanks. I have another question: I have a mapping task that uses bwa to pipe to samtools. I'd like bwa to use ~75% of the ${task.cpus} and samtools the rest, or something like that. So I'd have to do some math on ${task.cpus} and make 2 new variables. Where would I do that? Before the actual script?
Paolo Di Tommaso
@pditommaso
Mar 08 2018 08:56
2) there's already a nextflow config -flat that dump the config as dot separated attribute that's straightforward to parse, but I like the idea of a yaml / json output, a PR is welcome for that ! :)
Paolo Di Tommaso
@pditommaso
Mar 08 2018 09:06
@Shellfishgene you can do something like that
process foo {
  script:
  def cpus1 = task.cpus * .75 
  def cpus2 = task.cpus * .25 
  """
  bwa ... etc 
  """ 
}
obviously you need to manage rounding properly maybe using an helper method
Shellfishgene
@Shellfishgene
Mar 08 2018 09:12
This should work def cpu_bwa = Math.floor( task.cpus * 0.75) def cpu_samtools = task.cpus - cpu_bwa
Paolo Di Tommaso
@pditommaso
Mar 08 2018 09:13
sounds reasonable
Shellfishgene
@Shellfishgene
Mar 08 2018 09:14
thanks!
Paolo Di Tommaso
@pditommaso
Mar 08 2018 09:19
:+1:
Shellfishgene
@Shellfishgene
Mar 08 2018 12:36
Does the error No such variable: mean that there is an error in the script, or that the data does not come from a channel like it should? The error message references an output section, set id, file "${id}_sort.bam" into sorted_bam
Shellfishgene
@Shellfishgene
Mar 08 2018 13:15
Never mind, I figured out it needs to be file("${id}_sort.bam")
Phil Ewels
@ewels
Mar 08 2018 13:15
Thanks @pditommaso - the -flat thing is great, that's super easy to parse

1) not possible, the script execution depends on the config object, therefore it must be parsed before the script itself

I understand this - I want the config object and the script config though. So can still be done in order. I just want to see stuff that's defined in the pipeline script as well. eg. version

Paolo Di Tommaso
@pditommaso
Mar 08 2018 13:34
@ewels something can be done, open a ticket for that please
Phil Ewels
@ewels
Mar 08 2018 14:11
:+1:
Shawn Rynearson
@srynobio
Mar 08 2018 16:37
@pditommaso I wanted to get your thoughts on mixing aws-abatch and local executors within the same nf script. I did some limited testing and it seem that they can not be mixed, mainly due to the inclusion of -w s3://bucket to the command line, i.e. the local tasks try to use the s3 work space and fail.
Paolo Di Tommaso
@pditommaso
Mar 08 2018 17:25
likely it does not work, but ideally it should
you may want to open a ticket for this
Shawn Rynearson
@srynobio
Mar 08 2018 17:34
:)