These are chat archives for nextflow-io/nextflow

23rd
May 2019
Laurence E. Bernstein
@lebernstein
May 23 00:20

@olgabot That's really bizarre, because I do this:

set val(sampleName), file("${inFile}.bai") into filtered_bam_indexed_ch

And it's just fine.
OK. Now that I think about it, maybe it makes sense. Because [file(x) into y)] is the equivalent of [z into y] whereas [file x into y] is something else?

Jason Steen
@jasteen
May 23 00:48
can anyone offer some suggestions on the best way to trouble shoot why a pipeline wont resume correctly. I know i'm on a lustre filesystem, which is a known problem, but Im not 100% sure the best way to even go about starting to troubleshoot exactly what nextflow is looking for when it makes a decision about caching. so far cache 'deep' appears to not help.
Rad Suchecki
@rsuchecki
May 23 00:58
How about cache 'lenient' @jasteen ?
Jason Steen
@jasteen
May 23 01:04
for some reason my brain associates lenient with something bad. but i'll give it a try
Rad Suchecki
@rsuchecki
May 23 01:15
:-D
Jason Steen
@jasteen
May 23 01:59
@rsuchecki, it looks initially like lenient may help me out. its certainly cached a process that i've never had cache properly before. thanks heaps.
Stephen Kelly
@stevekm
May 23 02:12

@lebernstein

So.. what I was trying to do was set up a system where anything that happens will not stop good samples from being processed

Have you seen the 'finish' errorStrategy? https://www.nextflow.io/docs/latest/process.html#errorstrategy

we want 24-7 up time

what exactly did you have in mind in regards to this? It seems like Nextflow is not really designed for a continuously running task execution, maybe something like Celery would be better for that. Or were you alluding more to the error handling rather than literally running forever? For my pipeline, I just started with the basic workflow then added "fixes" as needed to avoid errors as they cropped up. We are also running in "production mode" though our data inputs are pretty standardized, and I did spend a solid ~year building and debugging and prepping to figure out what makes things break.

@lebernstein

Suddenly I realized my workflow is not behaving properly at all and I know why.
The issue has been discussed here but I wanted to know what is the simple/best/proper solution.
I have a normal workflow with processes A --> B then A + B --> C. (C uses the input from A and B). When I run multiple samples in parallel > there is no guarantee that when I run C I will get matching inputs from A and B, even though B requires A because channels are filled in order of when they happen to run. Therefore I get incorrect and non-deterministic results from C!
What is the proper way to handle this? I know they are all supposed to have the same sample name.

sounds like something I have dealt with before but gonna need to see a demo script to be sure I understand the situation exactly...

Jason Steen
@jasteen
May 23 02:21
if c uses the input from A and B, then you need to output A and B as a tuple with matching sample names and then join them for processing into C. nextflow is asynchronous.
Stephen Kelly
@stevekm
May 23 02:22
oh yeah when you say "C uses the input from A and B" you need to be pre-combining the data into a single channel input for C. Do not have multiple input channels in a process unless you absolutely know what you are doing
@all what is the method to do the opposite of a groupTuple() ?

if I have a channel like this

[ sampleID,  [file1.txt, file2.txt] ]

I want to convert it to output these items:

[ sampleID,  file1.txt ]
[ sampleID,  file2.txt ]
Jason Steen
@jasteen
May 23 02:29
unTuple? I dont think there is an operator that does it though. at least, not that I can immediately see from the docs
Stephen Kelly
@stevekm
May 23 02:30
I am thinking it might be 'transpose'?
@tobsecret isnt this what you did the other day?

also to make it worse, my channel actually looks like this:

[ sample1,  file1.txt ]
[ sample2,  [file1.txt, file2.txt] ]

so like I guess I first have to remap it to turn the single files into a list then transpose?

Jason Steen
@jasteen
May 23 02:43
seems like it should work that way.
Stephen Kelly
@stevekm
May 23 02:45
weird I get an error:
ERROR ~ Not a valid transpose element at index: 0 -- Offending tuple: (MuTect2, SeraCare-1to2-Positive_HapMap-B17-1267, SeraCare-1to2-Positive, HapMap-B17-1267, 10, [igv-regions.bed.0])
using .transpose(by: [0,1,2,3,4])
Jason Steen
@jasteen
May 23 02:46
I cant even get the example to run
channel.from(1,2,3,4) print as expected. but channel.from([1,2,3]) prints nothing.
am I doing something stupid?
Stephen Kelly
@stevekm
May 23 02:48
I think it might have to do with making the channel from a list
the extra brackets, sometimes I have to double-braket
Jason Steen
@jasteen
May 23 02:49
the transpose example prints nothing for me too
ahhh. I remember something about that
Stephen Kelly
@stevekm
May 23 02:49
this is the code snippet I am working on:
igv_regions_split.map { caller, comparisonID, tumorID, normalID, chunkLabel, bed_files ->
    // check if its a list or not
    def is_list = bed_files instanceof Collection
    if(is_list){
        return([ caller, comparisonID, tumorID, normalID, chunkLabel, bed_files ])
    } else {
        // make it a list
        def bed_list = [ bed_files ]
        return([ caller, comparisonID, tumorID, normalID, chunkLabel, bed_list ])
    }
}
.transpose(by: [0,1,2,3,4])
.subscribe { println "[igv_regions_split] ${it}" }
but yeah it still does not work
Jason Steen
@jasteen
May 23 02:58
I think either transpose or reduce are what you are after, but i'm a bit green on nextflow and groovy to work out the exact syntax.
Stephen Kelly
@stevekm
May 23 02:59
I agree transpose looks to be exactly it but I got no clue why it wont work :(
if this was Python I would just use yield...
Jason Steen
@jasteen
May 23 03:00
is there a groovy function that does the same thing?
otherwise you need to rethink the whole data structure. it took me ages to work out a channel combination that would work nicely for somatic calling.
Jason Steen
@jasteen
May 23 03:05
especially once you start splitting things into chunks for parallel processing.
Stephen Kelly
@stevekm
May 23 03:14
i mean it works fine if I am only emitting the file by itself
then I can just use 'flatten'
but in this case I need to maintain the identifiers
cause like first I split the targets .bed file down to ~100 line chunks to run the variant calling in parallel, but now I am trying to IGV snapshot some variants and I have to chunk the variant tables further cause IGV is sloooow
Jason Steen
@jasteen
May 23 04:45
have you seen igv.js yet? apparently its the future
Laurence E. Bernstein
@lebernstein
May 23 07:09

@stevekm Soo.. I can't remember why, but errorStrategy 'finish' didn't really handle errors the way I wanted (or wouldn't let me log like I wanted?) so I went with 'ignore' and an afterScript. It is not really possible for me to describe all aspects of this project because it interfaces with other software that does pre- and post- processing so I am not writing a workflow that handles 100% of everything we do.

When I said 24/7 I meant that I don't want errors shutting down whole runs in some obscure way that requires developer intervention. The way I have it set up now I can resubmit my "job" and it will only re-run samples that failed previously. I'm actually using another process to submit various workflows to Nextflow as data becomes available, hence 24/7, since they can come in around-the-clock.

As you said, it took you a year to figure out what makes things break, I don't have this information currently so I'm making a "catch all" kind of error handling. Or at least I hope I am.

Right now it's working pretty well. I'm not sending in bogus data to see it fail so I have no idea what it will and will not handle. Since I'm mostly debugging my workflows currently, all the failures are due to workflow issues not data issues (obviously I'm using known good data for comparison).

I think this other issue is related to the network because I seem to be having other network issues, so I'm just going to see if it persists. It is not really reproducible/deterministic. :(

Paolo Di Tommaso
@pditommaso
May 23 07:20
is there anybody in the community having a trace.txt file generated by NF with some execution metrics willing to share?
Rad Suchecki
@rsuchecki
May 23 07:26
sure, just any trace?
Paolo Di Tommaso
@pditommaso
May 23 07:26
yep whatever, the bigger the better
Javan_Okendo
@javanOkendo
May 23 07:27
I am new in nexflow .....can someone give me a good pointer
Paolo Di Tommaso
@pditommaso
May 23 07:27
you have good chances
Paolo Di Tommaso
@pditommaso
May 23 07:44
excellent! thanks!
Ola Tarkowska
@olatarkowska
May 23 08:48
Hello, I have just started using nextflow so please forgive me begginer question. Could I please ask how nextflow determines what should go to .command.err vs .out ?
Paolo Di Tommaso
@pditommaso
May 23 08:49
no sorry, we don't accept beginner
.. kidding :D
Ola Tarkowska
@olatarkowska
May 23 08:49
heheh
So I am getting unexpected output in .err
Paolo Di Tommaso
@pditommaso
May 23 08:49
/dev/stderr > .command.err
/dev/stdout > .command.out
this was an easy one :D
does it help ?
Ola Tarkowska
@olatarkowska
May 23 08:50
so you are saying that the tool I am using (fiji) has a wrong level set on the logging ?
Paolo Di Tommaso
@pditommaso
May 23 08:51
it's possible
Ola Tarkowska
@olatarkowska
May 23 08:51
yes I woudl expect that warning or error would go to stderr and info to stdout
Paolo Di Tommaso
@pditommaso
May 23 08:52
try to redirect stderr to stdout
command >out 2>&1
where out is supposed to be a file (I think)
therefore
command >/dev/stdout  2>&1
Ola Tarkowska
@olatarkowska
May 23 08:53
let me test, thanks for the hint
Paolo Di Tommaso
@pditommaso
May 23 08:53
you are welcome
Ola Tarkowska
@olatarkowska
May 23 09:27
it is strange that curl does the same
not sure anyway minor issue
Paolo Di Tommaso
@pditommaso
May 23 09:27
the same what?
Ola Tarkowska
@olatarkowska
May 23 09:27
is logged to command.err
Paolo Di Tommaso
@pditommaso
May 23 09:28
it makes sense because the output of curl is supposed to be the file content
therefore curl messages/info are send to stderr
Ola Tarkowska
@olatarkowska
May 23 09:28
the only command that goes to out is the one that java set logging to INFO
yeah curl is a bit confusing even with -o
Thanks I change logger in fiji
Francesco Strozzi
@fstrozzi
May 23 09:37
@evanfloden et al. just to follow up on the r5d.x24large instance issue with AWS Batch (recap: Batch was not spawning this type of instance, even if it is supporting the instance family). I received word from the AWS support, they were able to reproduce the problem and confirm there is an issue with AWS Batch, which causes Batch to not open SPOT requests when the maximum amount of CPU and memory for this type of instance is required by a job.
The support escalated the problem to the AWS Batch team, they should come back soon I hope.
Evan Floden
@evanfloden
May 23 10:27
Thanks Francesco, I think I have seen this before too. The error messages/warnings from the Batch side could really be improved. I don't want to think about how many hours I spent in Jan troubleshooting.
Stephen Kelly
@stevekm
May 23 14:15

I seem to be having other network issues

@lebernstein yeah this kind of stuff is definitely something to watch out for, in my experience so far easily 95%+ (not exagerrating) of run failures are the result of system and hardware failure. Especially because I am running on an HPC with GPFS, I have had massive amounts of issues directly tied to things like LDAP cacheing, storage device lag/failure, SLURM failures, OS environment issues, etc.. When your compute infrastructure is solid, then Nextflow runs very smoothly, but if your server setup is spotty, you notice it quickly, especially when you have 1000's of tasks in your pipeline.

@jasteen

have you seen igv.js yet? apparently its the future

I have not, thought I was just looking at Genoverse. But my users are already using whatever the client-side JS implementation of IGV is and get really frustrated with it. I threw together an IGV snapshot demo in Nextflow just yesterday here: https://github.com/stevekm/IGV-snapshot-nf

But yeah IGV is unbearably slow, it takes ~2s per snapshot, which is not tolerable when you have hundreds and thousands to do... So I am trying to split the snapshot regions down into smaller chunks to run in parallel.

What would be really cool is if I could get one of those JS genome viewers embedded into a self-contained HTML file with a tiny snippet of the data from the .bam included, just like how Plotly and R Markdown HTML output works, but I do not now enough about JS to be able to do that easily...

Alaa Badredine
@AlaaBadredine_twitter
May 23 14:24
anyway to pass the -N (Workflow notification) command to nextflow.config ?
Stephen Kelly
@stevekm
May 23 14:37
@AlaaBadredine_twitter I think what you want is described here: https://www.nextflow.io/docs/latest/mail.html#mail-configuration
@pditommaso any suggestions on my 'un-tuple' situation and issues with the 'transpose' operator?
Alaa Badredine
@AlaaBadredine_twitter
May 23 14:43
@stevekm thanks for the links but that's not what I mean :( There's a special command with Nextflow that allows to generate a report directly https://www.nextflow.io/docs/latest/mail.html#workflow-notification
just like in the picture by passing the command -N xxx@mail.com
Stephen Kelly
@stevekm
May 23 14:44

@pditommaso trying to go from something like this:

[ sample1,  file1.txt ]
[ sample2,  [file1.txt, file2.txt] ]

to this

[ sample1,  file1.txt ]
[ sample2,  file1.txt ]
[ sample2,  file2.txt]

code so far:

igv_regions_split.map { caller, comparisonID, tumorID, normalID, chunkLabel, bed_files ->
    // check if its a list or not
    def is_list = bed_files instanceof Collection
    if(is_list){
        return([ caller, comparisonID, tumorID, normalID, chunkLabel, bed_files ])
    } else {
        // make it a list
        def bed_list = [ bed_files ]
        return([ caller, comparisonID, tumorID, normalID, chunkLabel, bed_list ])
    }
}
.transpose(by: [0,1,2,3,4])
.subscribe { println "[igv_regions_split] ${it}" }

error:

ERROR ~ Not a valid transpose element at index: 0 -- Offending tuple: (MuTect2, SeraCare-1to2-Positive_HapMap-B17-1267, SeraCare-1to2-Positive, HapMap-B17-1267, 10, [igv-regions.bed.0])
Alaa Badredine
@AlaaBadredine_twitter
May 23 14:45
so I was wondering if we could define this parameter directly in the config
Stephen Kelly
@stevekm
May 23 14:45
@AlaaBadredine_twitter yea my configs do this
Alaa Badredine
@AlaaBadredine_twitter
May 23 14:45
oh alright, thanks I will check it out !
Alaa Badredine
@AlaaBadredine_twitter
May 23 14:48
nice ! looks nice, I will test them out, thanks @stevekm
Netsanet Gebremedhin
@gnetsanet
May 23 15:11

Hello Everyone.

I am using AWS Batch as executor. I have over 25GB of reference files that I do not want to make part of an image or download on the fly. I would like to put that data on an EBS volume in a custom AMI and 'mount' that path/volume on the docker containers. I would appreciate any pointers on how to go about doing that.

Netsanet Gebremedhin
@gnetsanet
May 23 18:16
in short how do I mount an AMI directory into a container @tbugfinder ?
Stephen Kelly
@stevekm
May 23 23:13
btw I figure out why my transpose was not working... the by: argument is the index of the lists to be broken apart... not the index of the keys to transpose on... oops
so I just had to use normal .transpose() and it works. It seems that it automatically un-groups all lists in the channel