These are chat archives for nextflow-io/nextflow

24th
Jul 2017
Ashley S Doane
@DoaneAS
Jul 24 2017 02:44 UTC
Hi Paolo and everyone, I'm getting an issue in my pipeline's last step where multiple samples are showing up in the same work directory and as input in the last script. I haven't been able to sort out why this is happening. Are there some common reasons for this I can check? The script is here in case anyone has a chance to take a quick look https://github.com/DoaneAS/atacflow/blob/master/main.nf, and the it's the last step called ataqc
@aunderwo mounting $HOME is the normal behavior, though it can be changed. see the singularity documentation
Mike Smoot
@mes5k
Jul 24 2017 04:24 UTC
Hi @DoaneAS you've got several inputs to your ataqc, but nothing guaranteeing that Sample is the same sample each time. The ataqc is triggered when there is something on each of the specified input channels, but not necessarily the same sample. You can't guarantee that the earlier processes output things in the correct order because they may be running concurrently and thus out of order. To fix this you'll need something like groupTuple on the input channels to ataqc to make sure that all input files are aligned with the same sample.
For what it's worth, I write almost all of my processes so that there's only one input channel and then I work to combine multiple input channels with different operators (and much debugging with view) such that the input channel is properly constructed. Probably overkill, but it helps me stay sane...
Shellfishgene
@Shellfishgene
Jul 24 2017 07:23 UTC
How does this warning come about: n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled (see the log file for details) ? It seems my tasks have been killed by the scheduler because they ran too long, but nf does not seem to notice this and just prints this warning every 5 minutes forever...
Paolo Di Tommaso
@pditommaso
Jul 24 2017 07:47 UTC
It comes from what you are saying. If your cluster hard kill the task, NF cannot recognise the termination condition, therefore print this warning message
Shellfishgene
@Shellfishgene
Jul 24 2017 07:57 UTC
But nf does notice it's missing from the qstat list? Because it does actually retry the tasks when they are killed.
Paolo Di Tommaso
@pditommaso
Jul 24 2017 07:59 UTC
actually you are right, you may want to upload somewhere the complete log so I can give it a look
Shellfishgene
@Shellfishgene
Jul 24 2017 08:12 UTC
I just remembered I have errorStrategy = { task.exitStatus == Integer.MAX_VALUE ? 'retry' : 'finish' } in my config, could that be the reason?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:13 UTC
no
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:26 UTC
um, seems something strange happened here. I need some to review it
could you please uploaded it as an issue here. It will help to keep track and interact about the problem
Shellfishgene
@Shellfishgene
Jul 24 2017 08:34 UTC
ok, opened an issue, thanks
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:35 UTC
great
Shellfishgene
@Shellfishgene
Jul 24 2017 10:06 UTC
@pditommaso I also saw that one process gave an error, it was a typo in the command. However, if the previous process failed, why did that run at all?
Ah, apparently it ran for one file, got confused by the retries.
Paolo Di Tommaso
@pditommaso
Jul 24 2017 10:08 UTC
please, let's continue the discussion on the issue thread
Shellfishgene
@Shellfishgene
Jul 24 2017 10:08 UTC
ok
Simone Baffelli
@baffelli
Jul 24 2017 13:02 UTC
Hello. Is it possible that sometimes nextflow fails to resume from the cache? It seems to be pretty random, from time to time some processes at random fail and get submitted again
Félix C. Morency
@fmorency
Jul 24 2017 13:19 UTC
@baffelli yeah I also have this issue sometime but can't seem to reproduce it in a consistent manner
Simone Baffelli
@baffelli
Jul 24 2017 13:19 UTC
Indeed, it just seems to be random. It goes away when I retry
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:19 UTC
if it happens, means there's some nondeterministic in some of your inputs
Simone Baffelli
@baffelli
Jul 24 2017 13:19 UTC
Could it be related with inconsistent hashing?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:20 UTC
I don't think so
Simone Baffelli
@baffelli
Jul 24 2017 13:20 UTC
I can't find anything non-deterministic
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:21 UTC
I have seen this happen on file stored on NFS with not synchronised clock time
Simone Baffelli
@baffelli
Jul 24 2017 13:23 UTC
I was using files on a sshfs mount before
Félix C. Morency
@fmorency
Jul 24 2017 13:23 UTC
we're using smb
Simone Baffelli
@baffelli
Jul 24 2017 13:24 UTC
ah true I moved to smb as well
atm our server is down and I'm working locally
but I still see it sporadically
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:25 UTC
sshfs !
Simone Baffelli
@baffelli
Jul 24 2017 13:26 UTC
?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:28 UTC
mostly kidding, never used sshsf to run a pipeline
Simone Baffelli
@baffelli
Jul 24 2017 13:30 UTC
one has to do with the infrastructure ;)
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:30 UTC
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
in what meaning ?
Simone Baffelli
@baffelli
Jul 24 2017 13:31 UTC
they don't think it is necessary/they are afraid it would complicate their workflow/they think it can't be trusted :(
but then, most of them work with very short timeseries of satellite data, while I need to process series of up to 500 images/day taken over the course of months
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:33 UTC
"they don't think it is necessary" what ?
"they think it can't be trusted" what ?
Simone Baffelli
@baffelli
Jul 24 2017 13:34 UTC
no really
I should record the conversation I had with my thesis supervisor
he thinks that some shell scripts is all I should need
and that using a workflow system would complicate my work :)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:35 UTC
ah, yes
of course, a classic argument :smile:
Simone Baffelli
@baffelli
Jul 24 2017 13:36 UTC
but I don't care and happily use nextflow. And try to evangelize my colleagues
which of course are surprised how much easier my life is :sunglasses:, especially when I want to change some parameters or do a parameter sweep
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:37 UTC
I can say that even my PI, when I first talk about NF idea some years ago, he said that was useless
now it's a NatBiotech paper :sunglasses:
Simone Baffelli
@baffelli
Jul 24 2017 13:38 UTC
I'm doing a lot of analyses I wouldn't have done without nextflow because they would have been terribily tedious
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:39 UTC
exactly
Simone Baffelli
@baffelli
Jul 24 2017 13:40 UTC
and for the type of work I'm doing the push concept is an amazing fit
I was going crazy before when I used makefile or snakemake
Félix C. Morency
@fmorency
Jul 24 2017 13:40 UTC
+1 to all of this
Simone Baffelli
@baffelli
Jul 24 2017 13:41 UTC
especially when I had to iteratively reprocess things using previous results and such
Félix C. Morency
@fmorency
Jul 24 2017 13:41 UTC
NF and Singularity were both game changer here
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:41 UTC
:clap: :clap: :clap:
Félix C. Morency
@fmorency
Jul 24 2017 13:41 UTC
Now the problem is I have to buy more storage because we are running too many studies and generating too much data
Simone Baffelli
@baffelli
Jul 24 2017 13:41 UTC
I only whish we had a better infrastructure here. We have a very powerful compute node, but I can't convince them to install anything remotely useful
Simone Baffelli
@baffelli
Jul 24 2017 13:42 UTC
that's a problem for us as well
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:42 UTC
A lot of Nextflow hype is in the air :D
Simone Baffelli
@baffelli
Jul 24 2017 13:42 UTC
backups as well...the PI didn't want to pay for it
and now the raid system is failing :)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:42 UTC
cool!
Simone Baffelli
@baffelli
Jul 24 2017 13:43 UTC
luckly I've saved my data somewhere else
otherwise I could have left my PhD
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:43 UTC
ahah
you have to pay your own backup #phdlife
Simone Baffelli
@baffelli
Jul 24 2017 13:44 UTC
I can only backup part of it unfortunately
but the funny thing is that we don't lack funds at all. they are just not being directed where they are most needed
Félix C. Morency
@fmorency
Jul 24 2017 13:44 UTC
we're using Crashplan. $10/month/computer for unlimited data
Simone Baffelli
@baffelli
Jul 24 2017 13:45 UTC
I've told them. but the uni policy is that one should not rely on external services
they want do develop everything in-house
and that costs a lot of time and money
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:47 UTC
wow, unlimited ?
even if you upload 5 PB ?
Félix C. Morency
@fmorency
Jul 24 2017 13:47 UTC
Yup.
But keep in mind that the bandwitdh is slow-ish
Uploading 5PB would take a very long time
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:48 UTC
I was betting that
Félix C. Morency
@fmorency
Jul 24 2017 13:48 UTC
@baffelli can't you just do it without asking with your own stuff?
Simone Baffelli
@baffelli
Jul 24 2017 13:48 UTC
that's what I'm doing
e.g using our uni unofficial gitlab instead of our labs ridiculous svn
Félix C. Morency
@fmorency
Jul 24 2017 13:49 UTC
what? svn is still a thing? :P
Simone Baffelli
@baffelli
Jul 24 2017 13:50 UTC
sadly :fearful:
I hate it
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:54 UTC
LOL
well, at least it's not CSV
Simone Baffelli
@baffelli
Jul 24 2017 13:56 UTC
well, if you mean cvs i believe you :) but csv files are fine :grin:
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:57 UTC
oops, I meant CVS :)
Simone Baffelli
@baffelli
Jul 24 2017 13:58 UTC
I hope never to have to use it
Phil Ewels
@ewels
Jul 24 2017 14:01 UTC

but the uni policy is that one should not rely on external services. they want do develop everything in-house

This is totally insane! :confounded:

Do they build their own computing hardware too? drive to work in a car made from components forged in their own garage?
(sorry, "preaching to the converted" here I'm sure)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:04 UTC
LOL
Simone Baffelli
@baffelli
Jul 24 2017 14:04 UTC
that's insane indeed. Exibit A: instead of using a commercial cloud storage system they had to deploy their own customized owncloud. took two years to launch
nobody uses it because its uglier than dropxbox
same with the new website, where they used some heavily customized CMS, it took several years to be ready and it's not to useful
Félix C. Morency
@fmorency
Jul 24 2017 14:06 UTC
nih syndrome?
Simone Baffelli
@baffelli
Jul 24 2017 14:07 UTC
I don't want to be seen as the one bashing his own institution (its's great in many regards) but sometimes I think having too many funds promotes this sort of stupidity
those sorts of behaviors you can only afford if you are very generously funded
Félix C. Morency
@fmorency
Jul 24 2017 14:15 UTC
I wish I was generously funded
Phil Ewels
@ewels
Jul 24 2017 14:16 UTC
First world problems.. We just have too much money..! :laughing:
haha, sorry
Simone Baffelli
@baffelli
Jul 24 2017 14:16 UTC
we just don't use it well imho
Phil Ewels
@ewels
Jul 24 2017 14:16 UTC
Yup, and you're surely not alone in that respect
We also have a new cloud infrastructure being built in Sweden
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:17 UTC
wow, cool!
Phil Ewels
@ewels
Jul 24 2017 14:17 UTC
despite having a new AWS node opening in Stockholm next year, plus lots of other stuff
Cool if it works well and launches on time.. ;)
Simone Baffelli
@baffelli
Jul 24 2017 14:17 UTC
same type of behavior I see here in switzerland
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:18 UTC
will it be an academic cloud ?
Simone Baffelli
@baffelli
Jul 24 2017 14:18 UTC
I guess that's more of a problem in state backed institutions
because you have a certain amount of funding guaranteed no matter what
Phil Ewels
@ewels
Jul 24 2017 14:21 UTC
@pditommaso - don't know much about it yet and there are a few different initiatives. Just found this though: https://www.sunet.se/tjanster/virtuell-server-i-molnet/ (in Swedish sorry)
plus other pages alongside for IaaS
Shellfishgene
@Shellfishgene
Jul 24 2017 14:25 UTC
We're not allowed to use cloud services anymore for legal reasons...
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:26 UTC
Germany, right?
Shellfishgene
@Shellfishgene
Jul 24 2017 14:30 UTC
Yes
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:32 UTC
Any data, or only clinical data?
Shellfishgene
@Shellfishgene
Jul 24 2017 14:33 UTC
Any data. Well, mostly personal data - but this includes 'meeting with boss' in Google calendar, for example. They may just block dropbox et al. at the firewall level.
Simone Baffelli
@baffelli
Jul 24 2017 14:36 UTC
that's ridiculous
so many resources wasted if you need to roll your own cloud for anything
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:37 UTC
Wow, anyhow better the Chinese model IMO (no foreign cloud)
Simone Baffelli
@baffelli
Jul 24 2017 14:37 UTC
indeed
Shellfishgene
@Shellfishgene
Jul 24 2017 14:37 UTC
We'll, they don't have much choice, the same rules apply to us as for all other government institutes. And it's quite ok that the police or tax people can't put your stuff on some cloud storage.
Simone Baffelli
@baffelli
Jul 24 2017 14:39 UTC
of course, but to apply it indiscriminately does not make too much sense
Shellfishgene
@Shellfishgene
Jul 24 2017 14:40 UTC
yes, science is different. That's a problem on all levels though, needing to supply 3 quotes for 'alternative products' when you do lab experiments is also not ideal.
Simone Baffelli
@baffelli
Jul 24 2017 14:46 UTC
that problem is solved here by having a threshold
you only need different quotes above a certain expense
Shellfishgene
@Shellfishgene
Jul 24 2017 14:50 UTC
We do too, but it's pretty low.
Running multiple instances on nf in the same dir is probably not recommended?
With different workflows.
Simone Baffelli
@baffelli
Jul 24 2017 14:52 UTC
I don't recommend that
I've tried once and it messed up the way the pipelines were resumed
Shellfishgene
@Shellfishgene
Jul 24 2017 14:53 UTC
Yeah, that does not make sense with all the log files and whatnot. I withdraw my question ;)
Simone Baffelli
@baffelli
Jul 24 2017 14:53 UTC
my solution was to copy everything in a new directory ;)
so I could keep on resuming while avoiding messing things up
Félix C. Morency
@fmorency
Jul 24 2017 17:25 UTC
/bin/bash: .command.sh: No such file or directory what could be the cause of this error? I'm trying to run a NF pipeline inside docker (cicd stuff) and I get this error
Paolo Di Tommaso
@pditommaso
Jul 24 2017 17:49 UTC
this is likely a problem with docker mounts
Félix C. Morency
@fmorency
Jul 24 2017 17:52 UTC
mmm anything particular I need to do?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 17:54 UTC
that's supposed to work, you can change in the task work dir and troubleshoot by running bash -x .command.run
Félix C. Morency
@fmorency
Jul 24 2017 18:10 UTC
right it works now! :)
needed to add the proper singularity bind paths
Paolo Di Tommaso
@pditommaso
Jul 24 2017 18:11 UTC
ok !
Ashley S Doane
@DoaneAS
Jul 24 2017 19:58 UTC
@mes5k thanks. What if you have inputs from several processes, that all relate to the same sample? Perhaps only one input should define the variable Sample, since it could end up being a different sample.
Mike Smoot
@mes5k
Jul 24 2017 21:44 UTC

Hi @DoaneAS here's what I'm suggesting:

Channel.from( [1, "file1"], [2, "file2"], [3, "file3"] ).set{ a }
Channel.from( [1, "result1"], [2, "result2"], [3, "result3"] ).set{ b }
Channel.from( [1, "whatever1"], [2, "whatever2"], [3, "whatever3"] ).set{ c }
Channel.from( [1, "something1"], [2, "something2"], [3, "something3"] ).set{ d }

a.mix(b)
 .mix(c)
 .mix(d)
 .groupTuple(sort: true)
 .view()
 .set{ inch }

process xzy {

    input:
    set val(sample), val(file_list) from inch

    output:
    stdout into ouch

    script:
    """
    echo ${sample} ${file_list}
    """
}

ouch.view()

Channels a, b, c, and d are the outputs of your upstream processes where each contains a tuple containing the sample id and a result. mix those channels to combine them into a single channel containing all the results, and then use groupTuple to group things by sample id. Then your process xyz has a single input channel where each result in file_list is properly associated with sample. Note that in my canned example val(file_list) is a valbecause my examples are just strings, but since I believe you have actual files you should have file(file_list)