Hi Paolo and everyone, I'm getting an issue in my pipeline's last step where multiple samples are showing up in the same work directory and as input in the last script. I haven't been able to sort out why this is happening. Are there some common reasons for this I can check? The script is here in case anyone has a chance to take a quick look https://github.com/DoaneAS/atacflow/blob/master/main.nf, and the it's the last step called
ataqc
@aunderwo mounting $HOME is the normal behavior, though it can be changed. see the singularity documentation
Hi @DoaneAS you've got several inputs to your
ataqc, but nothing guaranteeing that Sample is the same sample each time. The ataqc is triggered when there is something on each of the specified input channels, but not necessarily the same sample. You can't guarantee that the earlier processes output things in the correct order because they may be running concurrently and thus out of order. To fix this you'll need something like groupTuple on the input channels to ataqc to make sure that all input files are aligned with the same sample.
For what it's worth, I write almost all of my processes so that there's only one input channel and then I work to combine multiple input channels with different operators (and much debugging with
view) such that the input channel is properly constructed. Probably overkill, but it helps me stay sane...
How does this warning come about:
n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled (see the log file for details) ? It seems my tasks have been killed by the scheduler because they ran too long, but nf does not seem to notice this and just prints this warning every 5 minutes forever...
It comes from what you are saying. If your cluster hard kill the task, NF cannot recognise the termination condition, therefore print this warning message
But nf does notice it's missing from the qstat list? Because it does actually retry the tasks when they are killed.
actually you are right, you may want to upload somewhere the complete log so I can give it a look
I just remembered I have
errorStrategy = { task.exitStatus == Integer.MAX_VALUE ? 'retry' : 'finish' } in my config, could that be the reason?
no
um, seems something strange happened here. I need some to review it
could you please uploaded it as an issue here. It will help to keep track and interact about the problem
ok, opened an issue, thanks
great
@pditommaso I also saw that one process gave an error, it was a typo in the command. However, if the previous process failed, why did that run at all?
Ah, apparently it ran for one file, got confused by the retries.
please, let's continue the discussion on the issue thread
ok
Hello. Is it possible that sometimes nextflow fails to resume from the cache? It seems to be pretty random, from time to time some processes at random fail and get submitted again
@baffelli yeah I also have this issue sometime but can't seem to reproduce it in a consistent manner
Indeed, it just seems to be random. It goes away when I retry
if it happens, means there's some nondeterministic in some of your inputs
Could it be related with inconsistent hashing?
I don't think so
I can't find anything non-deterministic
I have seen this happen on file stored on NFS with not synchronised clock time
I was using files on a sshfs mount before
we're using smb
ah true I moved to smb as well
atm our server is down and I'm working locally
but I still see it sporadically
sshfs !
?
mostly kidding, never used sshsf to run a pipeline
one has to do with the infrastructure ;)
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
my colleagues are mostly skeptical about the whole pipeline/reproducibility story
in what meaning ?
they don't think it is necessary/they are afraid it would complicate their workflow/they think it can't be trusted :(
but then, most of them work with very short timeseries of satellite data, while I need to process series of up to 500 images/day taken over the course of months
"they don't think it is necessary" what ?
"they think it can't be trusted" what ?
no really
I should record the conversation I had with my thesis supervisor
he thinks that some shell scripts is all I should need
and that using a workflow system would complicate my work :)
ah, yes
of course, a classic argument :smile:
but I don't care and happily use nextflow. And try to evangelize my colleagues
which of course are surprised how much easier my life is :sunglasses:, especially when I want to change some parameters or do a parameter sweep
I can say that even my PI, when I first talk about NF idea some years ago, he said that was useless
now it's a NatBiotech paper :sunglasses:
I'm doing a lot of analyses I wouldn't have done without nextflow because they would have been terribily tedious
exactly
and for the type of work I'm doing the push concept is an amazing fit
I was going crazy before when I used makefile or snakemake
+1 to all of this
especially when I had to iteratively reprocess things using previous results and such
NF and Singularity were both game changer here
:clap: :clap: :clap:
Now the problem is I have to buy more storage because we are running too many studies and generating too much data
I only whish we had a better infrastructure here. We have a very powerful compute node, but I can't convince them to install anything remotely useful
that's a problem for us as well
A lot of Nextflow hype is in the air :D
backups as well...the PI didn't want to pay for it
and now the raid system is failing :)
cool!
luckly I've saved my data somewhere else
otherwise I could have left my PhD
ahah
you have to pay your own backup #phdlife
I can only backup part of it unfortunately
but the funny thing is that we don't lack funds at all. they are just not being directed where they are most needed
we're using Crashplan. $10/month/computer for unlimited data
I've told them. but the uni policy is that one should not rely on external services
they want do develop everything in-house
and that costs a lot of time and money
wow, unlimited ?
even if you upload 5 PB ?
Yup.
But keep in mind that the bandwitdh is slow-ish
Uploading 5PB would take a very long time
I was betting that
@baffelli can't you just do it without asking with your own stuff?
that's what I'm doing
e.g using our uni unofficial gitlab instead of our labs ridiculous svn
what? svn is still a thing? :P
sadly :fearful:
I hate it
LOL
well, at least it's not CSV
well, if you mean cvs i believe you :) but csv files are fine :grin:
oops, I meant CVS :)
I hope never to have to use it
but the uni policy is that one should not rely on external services. they want do develop everything in-house
This is totally insane! :confounded:
Do they build their own computing hardware too? drive to work in a car made from components forged in their own garage?
(sorry, "preaching to the converted" here I'm sure)
LOL
that's insane indeed. Exibit A: instead of using a commercial cloud storage system they had to deploy their own customized owncloud. took two years to launch
nobody uses it because its uglier than dropxbox
same with the new website, where they used some heavily customized CMS, it took several years to be ready and it's not to useful
nih syndrome?
I don't want to be seen as the one bashing his own institution (its's great in many regards) but sometimes I think having too many funds promotes this sort of stupidity
those sorts of behaviors you can only afford if you are very generously funded
I wish I was generously funded
First world problems.. We just have too much money..! :laughing:
haha, sorry
we just don't use it well imho
Yup, and you're surely not alone in that respect
We also have a new cloud infrastructure being built in Sweden
wow, cool!
despite having a new AWS node opening in Stockholm next year, plus lots of other stuff
Cool if it works well and launches on time.. ;)
same type of behavior I see here in switzerland
will it be an academic cloud ?
I guess that's more of a problem in state backed institutions
because you have a certain amount of funding guaranteed no matter what
@pditommaso - don't know much about it yet and there are a few different initiatives. Just found this though: https://www.sunet.se/tjanster/virtuell-server-i-molnet/ (in Swedish sorry)
plus other pages alongside for IaaS
We're not allowed to use cloud services anymore for legal reasons...
Germany, right?
Yes
Any data, or only clinical data?
Any data. Well, mostly personal data - but this includes 'meeting with boss' in Google calendar, for example. They may just block dropbox et al. at the firewall level.
that's ridiculous
so many resources wasted if you need to roll your own cloud for anything
Wow, anyhow better the Chinese model IMO (no foreign cloud)
indeed
We'll, they don't have much choice, the same rules apply to us as for all other government institutes. And it's quite ok that the police or tax people can't put your stuff on some cloud storage.
of course, but to apply it indiscriminately does not make too much sense
yes, science is different. That's a problem on all levels though, needing to supply 3 quotes for 'alternative products' when you do lab experiments is also not ideal.
that problem is solved here by having a threshold
you only need different quotes above a certain expense
We do too, but it's pretty low.
Running multiple instances on nf in the same dir is probably not recommended?
With different workflows.
I don't recommend that
I've tried once and it messed up the way the pipelines were resumed
Yeah, that does not make sense with all the log files and whatnot. I withdraw my question ;)
my solution was to copy everything in a new directory ;)
so I could keep on resuming while avoiding messing things up
/bin/bash: .command.sh: No such file or directory what could be the cause of this error? I'm trying to run a NF pipeline inside docker (cicd stuff) and I get this error
this is likely a problem with docker mounts
mmm anything particular I need to do?
that's supposed to work, you can change in the task work dir and troubleshoot by running
bash -x .command.run
right it works now! :)
needed to add the proper singularity bind paths
ok !
@mes5k thanks. What if you have inputs from several processes, that all relate to the same sample? Perhaps only one input should define the variable
Sample, since it could end up being a different sample.
Hi @DoaneAS here's what I'm suggesting:
Channel.from( [1, "file1"], [2, "file2"], [3, "file3"] ).set{ a }
Channel.from( [1, "result1"], [2, "result2"], [3, "result3"] ).set{ b }
Channel.from( [1, "whatever1"], [2, "whatever2"], [3, "whatever3"] ).set{ c }
Channel.from( [1, "something1"], [2, "something2"], [3, "something3"] ).set{ d }
a.mix(b)
.mix(c)
.mix(d)
.groupTuple(sort: true)
.view()
.set{ inch }
process xzy {
input:
set val(sample), val(file_list) from inch
output:
stdout into ouch
script:
"""
echo ${sample} ${file_list}
"""
}
ouch.view()Channels a, b, c, and d are the outputs of your upstream processes where each contains a tuple containing the sample id and a result. mix those channels to combine them into a single channel containing all the results, and then use groupTuple to group things by sample id. Then your process xyz has a single input channel where each result in file_list is properly associated with sample. Note that in my canned example val(file_list) is a valbecause my examples are just strings, but since I believe you have actual files you should have file(file_list)