DAGs look great!
Is there a recommended way to debug when an ignite cluster node isn't being assigned jobs?
Is there a recommended way to debug when an ignite cluster node isn't being assigned jobs?
When the master node starts up, the
.node-nextflow.log on the snubbed machine includes lines like:May-10 16:51:07.332 [exchange-worker-#69%nextflow%] INFO o.a.i.i.p.c.GridCachePartitionExchangeManager - Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, node=96c7d4b0-dc9b-4a9d-9a50-1b87bd4c512e
.but there are still plenty of jobs in the queue.
The full logs are here (but any pointers to debug myself would be great)
Oh, the "Skipping rebalancing (nothing scheduled)" is expected, as the worker node has no jobs scheduled when it sees the master come online.
You could enable a more detailed logging on the daemon side by using
-trace nextflow.daemon.CustomStealingCollisionSpi
However I guess that's on the nodes there aren't enough mem/cpus
Could it be that?
I don't think so. Would that be reported with --trace?
This
Perfect
and this
-trace is on the master node, right?
nope, on the daemon(s)
hmmm. I get
$ nextflow node -trace nextflow.daemon.CustomStealingCollisionSpi
Unknown option: -trace -- Check the available commands and options and syntax with 'help'
um, it should be
nextflow -trace ... node
silly me, sorry.
no pb!
There seem to be plenty of resources available on the lonely node:
TRACE n.daemon.CustomStealingCollisionSpi - Node `localhost` resources > cpus: 12 (12) - mem: 15.6 GB (942.5 MB) - active: 0 - waiting: 34
Oh, wait. Free mem < 1GB
That's it.
The JVM is consuming the remaining 15GB ?
No, it's being used as disk cache
oh
it's not too much ?
I don't think I have control over disk cache. My understanding is that memory used as cache is freed up as soon as any other application asks for it. I'd suggest that the free memory calculation should not include cache (but I'm not sure how to isolate that out).
you may be right, please report an issue at your convenience
Will do. Thanks Paolo!
Welcome!
In the meantime, a restart on the node is an easy fix :)
:)
hola, if I want to store some data directly from nextflow in json, can I use the groovy json capabilities somehow?
so i do not have to write an other script outside my flow
That's better: http://i.imgur.com/YBreC7g.png
@szilvajuhos Nextflow does not implement any special json feature, however it includes the excellent Groovy Json library. Have a look to http://groovy-lang.org/json.html
@robsyme A lot!
A fix faster than restarting:
# sync && echo 3 > /proc/sys/vm/drop_caches
ah, nice
to remember
@szilvajuhos I think you are looking for JsonBuilder
that is ok, I just do not know how to mix nf and groovy
nothing special
NF is a groovy script
so just import it at the top of your script and then use it as you would do in a groovy script
that is what I have expected, but likely I am making a simple error, like even if the whole "script" is nothing else but a single line of import statement, I am getting this:
@mes5k That a lot for your PR! I've commented it. By the way do you have a screenshot of the produced HTML ?
obs_test.nf: 1: unable to resolve class groovy.json
@ line 1, column 1.
import groovy.json
^
@ line 1, column 1.
import groovy.json
^
import groovy.json.*
like in java, the only difference is that the semicolon is not required
yes, as soon as I pasted, saw that the star is missing
yep
kul, now it works, thanks for sorting out
:+1:
@robsyme I've uploaded
0.19.0-SNAPSHOT with a patch for #150, you may want to give it a try
That's a screenshot of a nextflow DAG rendered with Cytoscape.js. It's a bit crooked in places because I adjusted the positioning of the nodes manually. You can zoom in/out on the actual html so it won't be as small in real life.
Not bad
do you think it could be useful ?
Yes. Of course, what I'd really like to see is the visualization updated (e.g. nodes changing colors) dynamically as the pipeline progresses. Then serve that page somewhere so that people can easily visualize their progress. But I understand that's a bunch more work!
Yes, that would be very cool, but at this time it seems not feasible
Understood.
however there's a plan to introduce an execution API that could be used to stream dag/timeline interactive rendering
It seems like TraceObserver already gets us most of the way there, no? The roadblock as I understand it is that DAG is only created after the pipeline runs.
I was wondering if it can be improved using D3
Any idea?
It seems like TraceObserver already gets us most of the way there, no? The roadblock as I understand it is that DAG is only created after the pipeline runs.
yes
the biggest problem currently is that there isn't a daemon serving the output
Yeah, D3 would work too. I just used Cytoscape because I've got history with them. D3's model for animation with data entering and exiting the visualization would be a natural fit for a dynamic visualization.
gotta run, but I'm going to experiment with a few things when I get a chance
I won't seems ungrateful and I'm really happy with your PR, but I would prefer to give a try to D3 before merging it
Bare bones D3 has been added. It still needs some work to make it look nice, but that's not something I can work on at this point.