These are chat archives for dgraph-io/dgraph

1st
Mar 2016
Manish R Jain
@manishrjain
Mar 01 2016 00:14
Simple end to end test works!
curl 127.0.0.1:8080/query -XPOST -d 'mutation {set { <ryan> <name> "reynolds" . }} query { me(_xid_: ryan) {name}}'
Response: {"_root_":[{"_uid_":"0xc6e142f7e6a7899e","name":"reynolds"}],"server_latency":{"json":"29.598µs","parsing":"53.567481ms","processing":"65.747µs","total":"53.665064ms"}}
Manish R Jain
@manishrjain
Mar 01 2016 01:25
And tested over multiple workers as well. It works!
All the way from uid assignment at worker 0, to distribution of sub-queries to all workers.
And it can all be done in the same GraphQL query.
Irrespective of which server is hit with the query. The server would automatically send out the right network calls to corresponding workers.
Super excited! :sparkles:
Manish R Jain
@manishrjain
Mar 01 2016 03:57
@ashwin95r When you get a chance, can you have a look at my change?
Manish R Jain
@manishrjain
Mar 01 2016 04:16
Btw, some comments looking at the code. Always, do defer wg.Done() instead of calling it at the end.
Manish R Jain
@manishrjain
Mar 01 2016 04:46
Looks like malloc and garbage collector is the slowest thing here:
This message was deleted
This message was deleted
(pprof) top10
4.19mins of 7.46mins total (56.23%)
Dropped 569 nodes (cum <= 0.75mins)
Showing top 10 nodes out of 25 (cum >= 2.78mins)
flat flat% sum% cum cum%
1.71mins 22.86% 22.86% 4.03mins 54.02% runtime.scanobject
1.16mins 15.62% 38.48% 1.16mins 15.62% runtime.heapBitsForObject
0.95mins 12.77% 51.25% 1.21mins 16.16% runtime.greyobject
0.19mins 2.56% 53.81% 3.46mins 46.41% runtime.mallocgc
0.07mins 0.88% 54.69% 1.08mins 14.53% runtime.mapassign1
0.04mins 0.48% 55.17% 3.82mins 51.17% runtime.systemstack
0.03mins 0.35% 55.52% 0.92mins 12.36% runtime.gcDrain
0.02mins 0.26% 55.78% 1.42mins 18.98% github.com/dgraph-io/dgraph/rdf.Parse
0.02mins 0.23% 56.01% 0.92mins 12.29% runtime.newobject
0.02mins 0.22% 56.23% 2.78mins 37.25% runtime.gcDrainN
Ashwin Ramesh
@ashwin95r
Mar 01 2016 05:17
Great news that it seems to work fine! :smile:
Manish R Jain
@manishrjain
Mar 01 2016 05:17
Yeah, feels awesome!
Distributed mutations working along with distributed queries -- that's the best :-).
Working on uid assigner now -- checking if we can optimize memory allocation.
Ashwin Ramesh
@ashwin95r
Mar 01 2016 05:19
So that the garbage collector is called less often. right?
Manish R Jain
@manishrjain
Mar 01 2016 05:19
Yeah, that's the idea. I'm looking at sync.Pool.
Doing memory profiling -- turns out rdf.Parse is the most expensive thing. Because, it creates a lexer object for each line.
Manish R Jain
@manishrjain
Mar 01 2016 07:02
Hey @ashwin95r , I made more changes based on more testing.
Can you please review the code again?
Ashwin Ramesh
@ashwin95r
Mar 01 2016 11:37
Done!
Manish R Jain
@manishrjain
Mar 01 2016 12:06
I don’t see any LGTM… @ashwin95r
Ashwin Ramesh
@ashwin95r
Mar 01 2016 19:09
Ah! sorry about that...I forgot