These are chat archives for nextflow-io/nextflow

31st
Jul 2017
Paolo Di Tommaso
@pditommaso
Jul 31 2017 06:51
@sergpolly that happens because NF handles UNKNWN as an error condition and assumes the job is not running any more
as consequence of that it tries look for the .exitcode file, since that file was not available after 4.5 mins (not 90 seconds) it discarded that job and submitted a new execution
this time it is working as expected
also in the log I read Execution is retried (1)
Paolo Di Tommaso
@pditommaso
Jul 31 2017 06:56
that means that was the first time that job was re-executed not for the 3rd time as you are reporting
Oskar Vidarsson
@oskarvid
Jul 31 2017 07:37
another thing that I feel should be enabled by default is the possibility to easily track which input file came from where. I have 8 input files for one tool, but no straight forward way of finding them unless I do it manually, looking up the hashes etc. It makes debugging tedious. A folder like "inputs" in work/xy/hash/ would be helpful with just a soft link to the files so it's easy to go through them.
Paolo Di Tommaso
@pditommaso
Jul 31 2017 07:39
the idea is to add that info a in the provenance report
Oskar Vidarsson
@oskarvid
Jul 31 2017 07:39
alright, sounds good
Bili Dong
@qobilidop
Jul 31 2017 08:10
Hi, I’m wondering if I could do nextflow run some/subdirectory/main.nf. When I tried, some/subdirectory is interpreted as a GitHub repository, which is not what I intended.
Paolo Di Tommaso
@pditommaso
Jul 31 2017 08:10
make sure the path exist
Bili Dong
@qobilidop
Jul 31 2017 08:11
Let me try again
Paolo Di Tommaso
@pditommaso
Jul 31 2017 08:11
if the path does not exist NF will try to pull a GH repo with that name
Bili Dong
@qobilidop
Jul 31 2017 08:11
I see
Bili Dong
@qobilidop
Jul 31 2017 08:17
so my nextflow run some/subdirectory/main.nf is in the script part of a process. I tried the command directly and it works. but in a process it doesn’t.
Ah I see
The working path changed
Paolo Di Tommaso
@pditommaso
Jul 31 2017 08:17
the process is executed in its own work dir
Bili Dong
@qobilidop
Jul 31 2017 08:18
Thanks, I understand the issue now.
workflow.projectDir solves my problem :)
Paolo Di Tommaso
@pditommaso
Jul 31 2017 08:20
;)
Brian Reichholf
@breichholf
Jul 31 2017 09:30
I'm getting a weird error I'm having difficulties interpreting:
ERROR ~ No signature of method: makeIndex() is applicable for argument types: (_nf_script_dd52da22$_run_closure5) values: [_nf_script_dd52da22$_run_closure5@6d2260db]

 -- Check script 'main.nf' at line: 100 or see '.nextflow.log' file for more details
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:31
what's at line 100 ?
Brian Reichholf
@breichholf
Jul 31 2017 09:34
Argh, never mind.... ignore my stupidity. For some inexplicable reason I wrote prepare makeIndex instead of process makeIndex 😅
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:34
very well!
Summer release happiness !
Oskar Vidarsson
@oskarvid
Jul 31 2017 09:51
I've changed the way I define the input files, I take the file path from a tsv file, and this upsets the "set pair_id" stuff I need to do in order to get the correct pairs as input for MergeBamAlignments, how do I "set pair_id" when I take the input files from a tsv? just doing "set pair_id, each line from etc" doesn't work. If I do I get the error message "ERROR ~ No such variable: from "
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:53
of you will specify it in the tsv file or you need to extract from the file names somehow
Oskar Vidarsson
@oskarvid
Jul 31 2017 09:53
so fastq.basename something?
because I need the tsv to create the read group correctly
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:54
it looks to me the simplest way is to add a column contain the pair_id
could that be doable ?
Oskar Vidarsson
@oskarvid
Jul 31 2017 09:55
in the tsv? sure, what should it contain though?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:55
what should it contain though?
Oskar Vidarsson
@oskarvid
Jul 31 2017 09:55
like pair_1, pair_2 etc for each pair?
and how would I change the NF code to use that?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:56
umm, should not be something like
pair_id  /path/../pair_1   /path/../pair_2
?
Oskar Vidarsson
@oskarvid
Jul 31 2017 09:58
This is what itit looks like now NA12878 H7LJGBBXX 1 test_R1.fastq test_R2.fastq LIB1 ILLUMINA
so just add "pair_id" as a column?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 09:59
which one is supposed to be the pair_id ?
Oskar Vidarsson
@oskarvid
Jul 31 2017 10:01
I don't know what pair_id should look like to begin with, but everything on that row is used as input, it's indexed and put in place with ${item[n]}
each row is unique
the next row could look like NA12878 T7LJGBBNG 1 test2_R1.fastq test2_R2.fastq LIB2 ILLUMINA
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:03
well, it's not mandatory
Oskar Vidarsson
@oskarvid
Jul 31 2017 10:03
read_id isn't? how do I solve the pairing issue though?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:03
when using fromFilePairs NF simply infers the file pair common prefix eg. test or test2 in your example
you can use whatever you want or even omit it
where are you reading that file?
I mean the line ?
and MergeBamAlignment needs to take the correct pairs as input, so that lane 1 of the mapped and unmapped are merged, instead i.e lane 1 and 2
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:06
I see that is an invalid syntax
Oskar Vidarsson
@oskarvid
Jul 31 2017 10:06
it intetionally is
intentionally
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:06
ok
basically you want execute that processes for each read pair in the tsv file, right
Oskar Vidarsson
@oskarvid
Jul 31 2017 10:07
yep
the issue is how I tell MergeBamAlignment to take the correct pairs as input
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:08
so the logic is
1) read the file
2) parse as a tsv
3) extract the file columns
4) transform the string path to file objects
5) create a channel producing that file pairs
for example
Oskar Vidarsson
@oskarvid
Jul 31 2017 10:09
yes, and it works
except for the file pair thing
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:11
a few second s
Paolo Di Tommaso
@pditommaso
Jul 31 2017 10:18
I was fighting with tab and spaces with my editor .. :/
anyhow
Channel.fromPath('tsv')
       .splitCsv(sep:'\t')
       .map { cols -> tuple(file(cols[3]), file(cols[4])) }
       .into { tsv_ch1; tsv_ch2 }


process foo {
  input: 
  file reads from tsc_ch1

  ..
}
something like the above
Oskar Vidarsson
@oskarvid
Jul 31 2017 11:45
I made it work with some edits, but now MergeBamAlignments doesn't work, it doesn't even give an error message that something is wrong, it just stops after FastqToSam, I suppose it's because I'm not using "set paid_id" on bwa or fastqtosam anymore, but MergeBamAlignment depended on it, so how do I tell MergeBamAlignment which pairs to take as input? I can't use "set pair_id, file(reads) from tsv_ch1" because I made it read the entire row so I could index it and use it to populate the strings for the read group creation.
Maxime Garcia
@MaxUlysse
Jul 31 2017 12:10
@pditommaso I'm trying the new pulling of singularity URIs thing, and I can pull without problems docker URLs, but I didn't have any luck with shub URLs, I'm using the same URL as when I'm pulling it with singularity, but it doesn't seems to work with Nextflow
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:11
Error message?
Maxime Garcia
@MaxUlysse
Jul 31 2017 12:11
"Missing singularity file for image shub://1496"
or "Missing singularity file for image shub://MaxUlysse/compile-beamer"
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:13
That are supposed to be valid URIs?
Maxime Garcia
@MaxUlysse
Jul 31 2017 12:14
yes
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:14
I will give a try
Maxime Garcia
@MaxUlysse
Jul 31 2017 12:14
if I do singularity pull shub://MaxUlysse/compile-beamer, it does work
the 1496 is just the id of the image
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:22
@oskarvid I have the feeling that you didn't get how NF channels work
Oskar Vidarsson
@oskarvid
Jul 31 2017 12:22
I'm sure you're correct
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:23
A channel emits one or more elements, each element is a tuple (ie an unmodifiable array) which contains one or more items
The structure of a tuple MUST match the process input to which is attached
For single item tuple you can use file or val
When a tuple contains more than one item you need to use a set, specifing the type of the nested items
Oskar Vidarsson
@oskarvid
Jul 31 2017 12:32
I appreciate your help
groovy and NF is a bit foreign to me, the syntax and underlying assumptions
Paolo Di Tommaso
@pditommaso
Jul 31 2017 12:48
I'm sure there are much more complex things in your work
Paolo Di Tommaso
@pditommaso
Jul 31 2017 13:05
@MaxUlysse It looks like a singularity bug
try this
singularity pull --name $PWD/foo.img shub://MaxUlysse/compile-beamer
Maxime Garcia
@MaxUlysse
Jul 31 2017 13:14
@pditommaso this command is not working
But singularity pull --name foo.img shub://MaxUlysse/compile-beamer works
Error message:
Progress ||----------------------------------|   0.0% ERROR Error downloading https://www.googleapis.com/download/storage/v1/b/singularity-hub-regional/o/singularity-hub-regional%2Fgithub.com%2FMaxUlysse%2Fcompile-beamer%2Fff2d43a31b04db4bee5c1aa53adeea9c634829b4%2F9ec05c5c6a1b970915bbc7a8c20836f0.img.gz?generation=1498659395821466&alt=media.
Do you have permission to write to /home/max/workspace/github/compile-beamer/home/max/workspace/github/compile-beamer?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 13:18
opened a bug singularityware/singularity#847
Maxime Garcia
@MaxUlysse
Jul 31 2017 13:18
Thanks a lot
Félix C. Morency
@fmorency
Jul 31 2017 14:28
@pditommaso Thanks for 0.25.3!
Paolo Di Tommaso
@pditommaso
Jul 31 2017 14:45
you are late, it's already 0.25.4 :)
Félix C. Morency
@fmorency
Jul 31 2017 14:46
haha yes I saw that :P
Paolo Di Tommaso
@pditommaso
Jul 31 2017 14:47
I guess you will appreciate the automatic pull of singularity images ;)
Paolo Di Tommaso
@pditommaso
Jul 31 2017 16:53
@MaxUlysse I've uploaded a new snapshot that should solve the problem NXF_VER=0.25.5-SNAPSHOT nextflow run .. etc
Félix C. Morency
@fmorency
Jul 31 2017 17:02
Yeah I need to investigate the deployment of a local singularity registry
Mike Smoot
@mes5k
Jul 31 2017 19:23

Hi @pditommaso have you ever seen an error like this before:

ERROR ~ Error executing process > 'runAugustus (0)'

Caused by:
  No such property: jobname for class: Script84071C1B35933246ACB28A846DB0F38B


Source block:
  #!/bin/bash

  for maxPieceSize in "200000", "100000", "150000"
  do
     ....

This is running on a slurm cluster and the runAugustus process has a bash template as the script:. I can post the full nextflow log if that helps, but I don't see anything else (apparently) associated with this error in the log.

Paolo Di Tommaso
@pditommaso
Jul 31 2017 19:25
weird, maybe a missing variable in the template
Mike Smoot
@mes5k
Jul 31 2017 19:27
I'll check!
You're right! a jobname variable is missing. Thanks for pointing me in the right direction!
Paolo Di Tommaso
@pditommaso
Jul 31 2017 19:30
:+1:
Mike Smoot
@mes5k
Jul 31 2017 19:31
I searched for jobname pretty much everywhere except the templates dir. Sigh. :)
Paolo Di Tommaso
@pditommaso
Jul 31 2017 19:31
LOL
Félix C. Morency
@fmorency
Jul 31 2017 20:02
Can I access $process.publishDir from inside the script: section?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 20:03
nope
or maybe task.publishDir
tho not sure
Félix C. Morency
@fmorency
Jul 31 2017 20:07
mmm it returns a PublishDir object
nextflow.processor.PublishDir(/tmp/Y/Results/results/Subject1, null, null, null, null)
Paolo Di Tommaso
@pditommaso
Jul 31 2017 20:08
oops ... I was positing in the wrong channel :grin:
Félix C. Morency
@fmorency
Jul 31 2017 20:08
Can I access the /tmp/Y/Results/results/Subject1 portion?
Paolo Di Tommaso
@pditommaso
Jul 31 2017 20:09
no, but instead of trying to access the publishDir property
you could parametrise it, and use the same in the process script
Félix C. Morency
@fmorency
Jul 31 2017 20:10
${task.publishDir.getPath()}
It works
Paolo Di Tommaso
@pditommaso
Jul 31 2017 20:10
OH !
unsupported hack !
:D
Félix C. Morency
@fmorency
Jul 31 2017 20:11
:P
Félix C. Morency
@fmorency
Jul 31 2017 20:16
mmm is there a way to pass env. variable to the container such as SINGULARITYENV_FOO singularity exec [inside_of_script:]
ah wait I can do it otherwise
Yeah nvm
Félix C. Morency
@fmorency
Jul 31 2017 20:22
Can I use a file "directory_name" as an output?
Félix C. Morency
@fmorency
Jul 31 2017 20:29
Yup