These are chat archives for nextflow-io/nextflow

24th
Jul 2017
Ashley S Doane
@DoaneAS
Jul 24 2017 02:44
Hi Paolo and everyone, I'm getting an issue in my pipeline's last step where multiple samples are showing up in the same work directory and as input in the last script. I haven't been able to sort out why this is happening. Are there some common reasons for this I can check? The script is here in case anyone has a chance to take a quick look https://github.com/DoaneAS/atacflow/blob/master/main.nf, and the it's the last step called ataqc
@aunderwo mounting $HOME is the normal behavior, though it can be changed. see the singularity documentation
Mike Smoot
@mes5k
Jul 24 2017 04:24
Hi @DoaneAS you've got several inputs to your ataqc, but nothing guaranteeing that Sample is the same sample each time. The ataqc is triggered when there is something on each of the specified input channels, but not necessarily the same sample. You can't guarantee that the earlier processes output things in the correct order because they may be running concurrently and thus out of order. To fix this you'll need something like groupTuple on the input channels to ataqc to make sure that all input files are aligned with the same sample.
For what it's worth, I write almost all of my processes so that there's only one input channel and then I work to combine multiple input channels with different operators (and much debugging with view) such that the input channel is properly constructed. Probably overkill, but it helps me stay sane...
Shellfishgene
@Shellfishgene
Jul 24 2017 07:23
How does this warning come about: n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled (see the log file for details) ? It seems my tasks have been killed by the scheduler because they ran too long, but nf does not seem to notice this and just prints this warning every 5 minutes forever...
Paolo Di Tommaso
@pditommaso
Jul 24 2017 07:47
It comes from what you are saying. If your cluster hard kill the task, NF cannot recognise the termination condition, therefore print this warning message
Shellfishgene
@Shellfishgene
Jul 24 2017 07:57
But nf does notice it's missing from the qstat list? Because it does actually retry the tasks when they are killed.
Paolo Di Tommaso
@pditommaso
Jul 24 2017 07:59
actually you are right, you may want to upload somewhere the complete log so I can give it a look
Shellfishgene
@Shellfishgene
Jul 24 2017 08:12
I just remembered I have errorStrategy = { task.exitStatus == Integer.MAX_VALUE ? 'retry' : 'finish' } in my config, could that be the reason?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:13
no
Shellfishgene
@Shellfishgene
Jul 24 2017 08:20
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:26
um, seems something strange happened here. I need some to review it
could you please uploaded it as an issue here. It will help to keep track and interact about the problem
Shellfishgene
@Shellfishgene
Jul 24 2017 08:34
ok, opened an issue, thanks
Paolo Di Tommaso
@pditommaso
Jul 24 2017 08:35
great
Shellfishgene
@Shellfishgene
Jul 24 2017 10:06
@pditommaso I also saw that one process gave an error, it was a typo in the command. However, if the previous process failed, why did that run at all?
Ah, apparently it ran for one file, got confused by the retries.
Paolo Di Tommaso
@pditommaso
Jul 24 2017 10:08
please, let's continue the discussion on the issue thread
Shellfishgene
@Shellfishgene
Jul 24 2017 10:08
ok
Simone Baffelli
@baffelli
Jul 24 2017 13:02
Hello. Is it possible that sometimes nextflow fails to resume from the cache? It seems to be pretty random, from time to time some processes at random fail and get submitted again
Félix C. Morency
@fmorency
Jul 24 2017 13:19
@baffelli yeah I also have this issue sometime but can't seem to reproduce it in a consistent manner
Simone Baffelli
@baffelli
Jul 24 2017 13:19
Indeed, it just seems to be random. It goes away when I retry
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:19
if it happens, means there's some nondeterministic in some of your inputs
Simone Baffelli
@baffelli
Jul 24 2017 13:19
Could it be related with inconsistent hashing?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:20
I don't think so
Simone Baffelli
@baffelli
Jul 24 2017 13:20
I can't find anything non-deterministic
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:21
I have seen this happen on file stored on NFS with not synchronised clock time
Simone Baffelli
@baffelli
Jul 24 2017 13:23
I was using files on a sshfs mount before
Félix C. Morency
@fmorency
Jul 24 2017 13:23
we're using smb
Simone Baffelli
@baffelli
Jul 24 2017 13:24
ah true I moved to smb as well
atm our server is down and I'm working locally
but I still see it sporadically
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:25
sshfs !
Simone Baffelli
@baffelli
Jul 24 2017 13:26
?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:28
mostly kidding, never used sshsf to run a pipeline
Simone Baffelli
@baffelli
Jul 24 2017 13:30
one has to do with the infrastructure ;)
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:30
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
in what meaning ?
Simone Baffelli
@baffelli
Jul 24 2017 13:31
they don't think it is necessary/they are afraid it would complicate their workflow/they think it can't be trusted :(
but then, most of them work with very short timeseries of satellite data, while I need to process series of up to 500 images/day taken over the course of months
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:33
"they don't think it is necessary" what ?
"they think it can't be trusted" what ?
Simone Baffelli
@baffelli
Jul 24 2017 13:34
no really
I should record the conversation I had with my thesis supervisor
he thinks that some shell scripts is all I should need
and that using a workflow system would complicate my work :)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:35
ah, yes
of course, a classic argument :smile:
Simone Baffelli
@baffelli
Jul 24 2017 13:36
but I don't care and happily use nextflow. And try to evangelize my colleagues
which of course are surprised how much easier my life is :sunglasses:, especially when I want to change some parameters or do a parameter sweep
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:37
I can say that even my PI, when I first talk about NF idea some years ago, he said that was useless
now it's a NatBiotech paper :sunglasses:
Simone Baffelli
@baffelli
Jul 24 2017 13:38
I'm doing a lot of analyses I wouldn't have done without nextflow because they would have been terribily tedious
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:39
exactly
Simone Baffelli
@baffelli
Jul 24 2017 13:40
and for the type of work I'm doing the push concept is an amazing fit
I was going crazy before when I used makefile or snakemake
Félix C. Morency
@fmorency
Jul 24 2017 13:40
+1 to all of this
Simone Baffelli
@baffelli
Jul 24 2017 13:41
especially when I had to iteratively reprocess things using previous results and such
Félix C. Morency
@fmorency
Jul 24 2017 13:41
NF and Singularity were both game changer here
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:41
:clap: :clap: :clap:
Félix C. Morency
@fmorency
Jul 24 2017 13:41
Now the problem is I have to buy more storage because we are running too many studies and generating too much data
Simone Baffelli
@baffelli
Jul 24 2017 13:41
I only whish we had a better infrastructure here. We have a very powerful compute node, but I can't convince them to install anything remotely useful
Simone Baffelli
@baffelli
Jul 24 2017 13:42
that's a problem for us as well
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:42
A lot of Nextflow hype is in the air :D
Simone Baffelli
@baffelli
Jul 24 2017 13:42
backups as well...the PI didn't want to pay for it
and now the raid system is failing :)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:42
cool!
Simone Baffelli
@baffelli
Jul 24 2017 13:43
luckly I've saved my data somewhere else
otherwise I could have left my PhD
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:43
ahah
you have to pay your own backup #phdlife
Simone Baffelli
@baffelli
Jul 24 2017 13:44
I can only backup part of it unfortunately
but the funny thing is that we don't lack funds at all. they are just not being directed where they are most needed
Félix C. Morency
@fmorency
Jul 24 2017 13:44
we're using Crashplan. $10/month/computer for unlimited data
Simone Baffelli
@baffelli
Jul 24 2017 13:45
I've told them. but the uni policy is that one should not rely on external services
they want do develop everything in-house
and that costs a lot of time and money
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:47
wow, unlimited ?
even if you upload 5 PB ?
Félix C. Morency
@fmorency
Jul 24 2017 13:47
Yup.
But keep in mind that the bandwitdh is slow-ish
Uploading 5PB would take a very long time
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:48
I was betting that
Félix C. Morency
@fmorency
Jul 24 2017 13:48
@baffelli can't you just do it without asking with your own stuff?
Simone Baffelli
@baffelli
Jul 24 2017 13:48
that's what I'm doing
e.g using our uni unofficial gitlab instead of our labs ridiculous svn
Félix C. Morency
@fmorency
Jul 24 2017 13:49
what? svn is still a thing? :P
Simone Baffelli
@baffelli
Jul 24 2017 13:50
sadly :fearful:
I hate it
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:54
LOL
well, at least it's not CSV
Simone Baffelli
@baffelli
Jul 24 2017 13:56
well, if you mean cvs i believe you :) but csv files are fine :grin:
Paolo Di Tommaso
@pditommaso
Jul 24 2017 13:57
oops, I meant CVS :)
Simone Baffelli
@baffelli
Jul 24 2017 13:58
I hope never to have to use it
Phil Ewels
@ewels
Jul 24 2017 14:01

but the uni policy is that one should not rely on external services. they want do develop everything in-house

This is totally insane! :confounded:

Do they build their own computing hardware too? drive to work in a car made from components forged in their own garage?
(sorry, "preaching to the converted" here I'm sure)
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:04
LOL
Simone Baffelli
@baffelli
Jul 24 2017 14:04
that's insane indeed. Exibit A: instead of using a commercial cloud storage system they had to deploy their own customized owncloud. took two years to launch
nobody uses it because its uglier than dropxbox
same with the new website, where they used some heavily customized CMS, it took several years to be ready and it's not to useful
Félix C. Morency
@fmorency
Jul 24 2017 14:06
nih syndrome?
Simone Baffelli
@baffelli
Jul 24 2017 14:07
I don't want to be seen as the one bashing his own institution (its's great in many regards) but sometimes I think having too many funds promotes this sort of stupidity
those sorts of behaviors you can only afford if you are very generously funded
Félix C. Morency
@fmorency
Jul 24 2017 14:15
I wish I was generously funded
Phil Ewels
@ewels
Jul 24 2017 14:16
First world problems.. We just have too much money..! :laughing:
haha, sorry
Simone Baffelli
@baffelli
Jul 24 2017 14:16
we just don't use it well imho
Phil Ewels
@ewels
Jul 24 2017 14:16
Yup, and you're surely not alone in that respect
We also have a new cloud infrastructure being built in Sweden
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:17
wow, cool!
Phil Ewels
@ewels
Jul 24 2017 14:17
despite having a new AWS node opening in Stockholm next year, plus lots of other stuff
Cool if it works well and launches on time.. ;)
Simone Baffelli
@baffelli
Jul 24 2017 14:17
same type of behavior I see here in switzerland
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:18
will it be an academic cloud ?
Simone Baffelli
@baffelli
Jul 24 2017 14:18
I guess that's more of a problem in state backed institutions
because you have a certain amount of funding guaranteed no matter what
Phil Ewels
@ewels
Jul 24 2017 14:21
@pditommaso - don't know much about it yet and there are a few different initiatives. Just found this though: https://www.sunet.se/tjanster/virtuell-server-i-molnet/ (in Swedish sorry)
plus other pages alongside for IaaS
Shellfishgene
@Shellfishgene
Jul 24 2017 14:25
We're not allowed to use cloud services anymore for legal reasons...
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:26
Germany, right?
Shellfishgene
@Shellfishgene
Jul 24 2017 14:30
Yes
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:32
Any data, or only clinical data?
Shellfishgene
@Shellfishgene
Jul 24 2017 14:33
Any data. Well, mostly personal data - but this includes 'meeting with boss' in Google calendar, for example. They may just block dropbox et al. at the firewall level.
Simone Baffelli
@baffelli
Jul 24 2017 14:36
that's ridiculous
so many resources wasted if you need to roll your own cloud for anything
Paolo Di Tommaso
@pditommaso
Jul 24 2017 14:37
Wow, anyhow better the Chinese model IMO (no foreign cloud)
Simone Baffelli
@baffelli
Jul 24 2017 14:37
indeed
Shellfishgene
@Shellfishgene
Jul 24 2017 14:37
We'll, they don't have much choice, the same rules apply to us as for all other government institutes. And it's quite ok that the police or tax people can't put your stuff on some cloud storage.
Simone Baffelli
@baffelli
Jul 24 2017 14:39
of course, but to apply it indiscriminately does not make too much sense
Shellfishgene
@Shellfishgene
Jul 24 2017 14:40
yes, science is different. That's a problem on all levels though, needing to supply 3 quotes for 'alternative products' when you do lab experiments is also not ideal.
Simone Baffelli
@baffelli
Jul 24 2017 14:46
that problem is solved here by having a threshold
you only need different quotes above a certain expense
Shellfishgene
@Shellfishgene
Jul 24 2017 14:50
We do too, but it's pretty low.
Running multiple instances on nf in the same dir is probably not recommended?
With different workflows.
Simone Baffelli
@baffelli
Jul 24 2017 14:52
I don't recommend that
I've tried once and it messed up the way the pipelines were resumed
Shellfishgene
@Shellfishgene
Jul 24 2017 14:53
Yeah, that does not make sense with all the log files and whatnot. I withdraw my question ;)
Simone Baffelli
@baffelli
Jul 24 2017 14:53
my solution was to copy everything in a new directory ;)
so I could keep on resuming while avoiding messing things up
Félix C. Morency
@fmorency
Jul 24 2017 17:25
/bin/bash: .command.sh: No such file or directory what could be the cause of this error? I'm trying to run a NF pipeline inside docker (cicd stuff) and I get this error
Paolo Di Tommaso
@pditommaso
Jul 24 2017 17:49
this is likely a problem with docker mounts
Félix C. Morency
@fmorency
Jul 24 2017 17:52
mmm anything particular I need to do?
Paolo Di Tommaso
@pditommaso
Jul 24 2017 17:54
that's supposed to work, you can change in the task work dir and troubleshoot by running bash -x .command.run
Félix C. Morency
@fmorency
Jul 24 2017 18:10
right it works now! :)
needed to add the proper singularity bind paths
Paolo Di Tommaso
@pditommaso
Jul 24 2017 18:11
ok !
Ashley S Doane
@DoaneAS
Jul 24 2017 19:58
@mes5k thanks. What if you have inputs from several processes, that all relate to the same sample? Perhaps only one input should define the variable Sample, since it could end up being a different sample.
Mike Smoot
@mes5k
Jul 24 2017 21:44

Hi @DoaneAS here's what I'm suggesting:

Channel.from( [1, "file1"], [2, "file2"], [3, "file3"] ).set{ a }
Channel.from( [1, "result1"], [2, "result2"], [3, "result3"] ).set{ b }
Channel.from( [1, "whatever1"], [2, "whatever2"], [3, "whatever3"] ).set{ c }
Channel.from( [1, "something1"], [2, "something2"], [3, "something3"] ).set{ d }

a.mix(b)
 .mix(c)
 .mix(d)
 .groupTuple(sort: true)
 .view()
 .set{ inch }

process xzy {

    input:
    set val(sample), val(file_list) from inch

    output:
    stdout into ouch

    script:
    """
    echo ${sample} ${file_list}
    """
}

ouch.view()

Channels a, b, c, and d are the outputs of your upstream processes where each contains a tuple containing the sample id and a result. mix those channels to combine them into a single channel containing all the results, and then use groupTuple to group things by sample id. Then your process xyz has a single input channel where each result in file_list is properly associated with sample. Note that in my canned example val(file_list) is a valbecause my examples are just strings, but since I believe you have actual files you should have file(file_list)