These are chat archives for dgraph-io/dgraph

2nd
Apr 2016
Manish R Jain
@manishrjain
Apr 02 2016 03:30
I think instead of Google Groups, we should probably use Discourse for the community
Prashanth Koppula
@koppula
Apr 02 2016 07:02
Agreed about the startups not scaling part. To that I would also add startups needing to forge their own path and not being afraid to try a different approach. And having a healthy maybe even an irrelevant attitude towards conventional wisdom.
Manish R Jain
@manishrjain
Apr 02 2016 07:04
đź‘Ť
Prashanth Koppula
@koppula
Apr 02 2016 07:19
Taking a look at Discourse and liking a lot of what I see
Prashanth Koppula
@koppula
Apr 02 2016 07:40
I've tried Slack, it has its advantages
Pawan Rawal
@pawanrawal
Apr 02 2016 07:42
slack is easily integrable with a bunch of stuff
and would become more configurable in the future, thats a big advantage.
Manish R Jain
@manishrjain
Apr 02 2016 07:47
I def like Slack better than Gitter. But, Gitter makes it easy for anyone to join it.
And its designed around open source projects.
Hopefully Gitter gets native apps, and fixes it's notification issues.
I don't think Discourse is alternative to Slack or Gitter. It's alternative to Google Groups though.
Pawan Rawal
@pawanrawal
Apr 02 2016 07:50
nowadays have seen some open source projects using slack too.
Pawan Rawal
@pawanrawal
Apr 02 2016 07:52
yes read this sometime back on Hackernews
Manish R Jain
@manishrjain
Apr 02 2016 07:52
Slack itself says, it's designed for communication within company. Not for large open source groups.
Though, we could try it. See how well it works.
Now they even have a Linux client.
Prashanth Koppula
@koppula
Apr 02 2016 07:54
There was a very interesting post by Brad Feld recently about Slack about overload. .let me dig it up
Pawan Rawal
@pawanrawal
Apr 02 2016 07:59
@manishrjain trying to load freebase film data
In the command go build . && ./uidassigner --numInstances 2 --instanceIdx 0 --rdfgzips $BENCHMARK_REPO/data/rdf-films.gz,$BENCHMARK_REPO/data/names.gz --uids ~/dgraph/uids/u0
what is dgraph/uids/u0 ?
Pawan Rawal
@pawanrawal
Apr 02 2016 08:05
got it
we have to create that, and thats where uids would be stored from xids probably
Manish R Jain
@manishrjain
Apr 02 2016 08:30
yup.
Also, you don’t have to build 2 instances. You can just build 1.
I can’t imagine having all the communication over emails anymore.
@koppula Re: the article.
Emails are a lot more heavy weight than real time communication via Slack.
Slack or Gitter make a distributed team feel lot more closer, and available.
Pawan Rawal
@pawanrawal
Apr 02 2016 08:35
ok
Manish R Jain
@manishrjain
Apr 02 2016 08:48
Hey @ashwin95r @koppula just sent you keybase invite.
Prashanth Koppula
@koppula
Apr 02 2016 08:48
Just pointing out another point of view. Emails definitely are more heavyweight. And I look at emails more for external communications
Manish R Jain
@manishrjain
Apr 02 2016 08:48
Would make it a bit easier to share confidential information / files.
Prashanth Koppula
@koppula
Apr 02 2016 08:49
Okay will take a look
Manish R Jain
@manishrjain
Apr 02 2016 08:49
@pawanrawal If you need a keybase invite as well, let me know.
Pawan Rawal
@pawanrawal
Apr 02 2016 08:54
yup, would love to have it :smile:
thanks, got it.
Manish R Jain
@manishrjain
Apr 02 2016 08:58
What should the discourse forum subhost be called?
talk.dgraph.io?
/all ^
or discuss.dgraph.io?
Pawan Rawal
@pawanrawal
Apr 02 2016 09:01
discuss.dgraph.io sounds better to me
Manish R Jain
@manishrjain
Apr 02 2016 09:01
Yeah, same feeling here :-)
Pawan Rawal
@pawanrawal
Apr 02 2016 09:03
i get some errors in the first pass
I am on a Mac , so I guess something could be wrong with my setup
Manish R Jain
@manishrjain
Apr 02 2016 09:04
what errors do you get?
Pawan Rawal
@pawanrawal
Apr 02 2016 09:05
goroutine 4627 [runnable, locked to thread]:
github.com/dgraph-io/dgraph/store/rocksdb._Cfunc_rocksdb_get(0x4d0e410, 0x4e29440, 0xc85406a0f0, 0xf, 0xc85406a150, 0xc859a5f7c8, 0x0)
    github.com/dgraph-io/dgraph/store/rocksdb/_obj/_cgo_gotypes.go:359 +0x45
github.com/dgraph-io/dgraph/store/rocksdb.(*DB).Get(0xc82002e048, 0xc82002e030, 0xc85406a0f0, 0xf, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/store/rocksdb/db.go:171 +0x303
github.com/dgraph-io/dgraph/store.(*Store).Get(0xc8200ba940, 0xc85406a0f0, 0xf, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/store/store.go:55 +0x85
github.com/dgraph-io/dgraph/posting.(*List).getPostingList(0xc85049e4e0, 0x44d4e88)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/posting/list.go:281 +0x281
github.com/dgraph-io/dgraph/posting.(*List).init(0xc85049e4e0, 0xc85406a0f0, 0xf, 0x10, 0xc8200ba940, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/posting/list.go:242 +0x270
github.com/dgraph-io/dgraph/posting.GetOrCreate(0xc85406a0f0, 0xf, 0x10, 0xc8200ba940, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/posting/lists.go:231 +0x24a
github.com/dgraph-io/dgraph/uid.GetOrAssign(0xc831ed7d81, 0x9, 0x0, 0x1, 0xc820209f00, 0x0, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/uid/assigner.go:214 +0x120
github.com/dgraph-io/dgraph/loader.(*state).assignUid(0xc820018690, 0xc831ed7d81, 0x9, 0x0, 0x0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/loader/loader.go:194 +0x170
github.com/dgraph-io/dgraph/loader.(*state).assignUidsOnly(0xc820018690, 0xc8200153b0)
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/loader/loader.go:220 +0x1a5
created by github.com/dgraph-io/dgraph/loader.AssignUids
    /Users/pawan/work/src/github.com/dgraph-io/dgraph/loader/loader.go:314 +0x289
Manish R Jain
@manishrjain
Apr 02 2016 09:05
Did you install rocksdb?
Pawan Rawal
@pawanrawal
Apr 02 2016 09:05
yes
Manish R Jain
@manishrjain
Apr 02 2016 09:06
And did you run the tests? go test ./...
Pawan Rawal
@pawanrawal
Apr 02 2016 09:08
yup they are good.
INFO[0029] Counters                                      ignored=0 len_cnq=10000 package=loader parsed=166097 pending=13000 processed=153097 read=176098
INFO[0030] Counters                                      ignored=0 len_cnq=9992 package=loader parsed=167680 pending=12989 processed=154691 read=177684
INFO[0030] List Merge counters                           added=0 clean=0 dirtysize=233304 mapsize=345631 merged=13402 package=posting pending=0
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fff94185286 m=2
last logs before the error begins
Pawan Rawal
@pawanrawal
Apr 02 2016 09:18
could be because of older version of rocksdb, I have 3.13 installed
Prashanth Koppula
@koppula
Apr 02 2016 09:23
@manishrjain discuss.dgraph.io sounds better
Pawan Rawal
@pawanrawal
Apr 02 2016 09:30
still get error after upgrading to 4.4 version
Manish R Jain
@manishrjain
Apr 02 2016 09:31
Which version of Go are you using?
@koppula Yup, setting it up with discuss.dgraph.io
Pawan Rawal
@pawanrawal
Apr 02 2016 09:32
ok so now go test fails after upgrading to rocksdb 4.4 for server.go
1.6
Manish R Jain
@manishrjain
Apr 02 2016 09:33
Which version of Go?
Pawan Rawal
@pawanrawal
Apr 02 2016 09:33
go 1.6
Manish R Jain
@manishrjain
Apr 02 2016 09:33
And the right Rocksdb, is v4.2
Pawan Rawal
@pawanrawal
Apr 02 2016 09:33
ok, let me try to install that.
Manish R Jain
@manishrjain
Apr 02 2016 09:46
Set up an email account with Sparkpost
So, Discourse can use that to send emails.
We get 100K emails per month for free.
Pawan Rawal
@pawanrawal
Apr 02 2016 10:06
still get the error, I think its a memory issue
my machine has 4 gigs of memory and uidassigner process is getting killed after a while
Manish R Jain
@manishrjain
Apr 02 2016 10:07
Set the stw to 2GB
What’s the error?
Pawan Rawal
@pawanrawal
Apr 02 2016 10:12
error is same as I sent above
Manish R Jain
@manishrjain
Apr 02 2016 10:13
pthread_create error?
Pawan Rawal
@pawanrawal
Apr 02 2016 10:13
yes
Manish R Jain
@manishrjain
Apr 02 2016 10:13
How many cores do you have?
Pawan Rawal
@pawanrawal
Apr 02 2016 10:14
two
Manish R Jain
@manishrjain
Apr 02 2016 10:14
Set the usable cores to 1.
There’s a flag for that.
Pawan Rawal
@pawanrawal
Apr 02 2016 10:14
ok
Manish R Jain
@manishrjain
Apr 02 2016 10:15
I think it’s possible that the number of goroutines are causing too many open files.
Can you share the complete commmand, and the log beginning 10 lines and the last 10 lines
Pawan Rawal
@pawanrawal
Apr 02 2016 10:17
okay, let me try with numCpu 1, and will share it
Manish R Jain
@manishrjain
Apr 02 2016 10:17
If it still doesn’t work, then we can bring up a GCE instance for you.
Pawan Rawal
@pawanrawal
Apr 02 2016 10:19
whats stw ?
Manish R Jain
@manishrjain
Apr 02 2016 10:20
run the command with —help
You’ll see the possible flags.
stw = stop the world
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:20
Hey @manishrjain , regarding RAFT, a new node can join the cluster only if it knows the ip of the leader at that time
not any node in the cluster
Pawan Rawal
@pawanrawal
Apr 02 2016 10:21
cool, didn't see the flag in the file so was wondering about it.
Manish R Jain
@manishrjain
Apr 02 2016 10:21
oh.. that’s funny
Btw, can you help @pawanrawal with a GCE instance set up?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:21
Should i set one up?
Manish R Jain
@manishrjain
Apr 02 2016 10:22
Yeah, you should have the rights to create a new instance.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:22
does he need some permissions to access the instance?
Manish R Jain
@manishrjain
Apr 02 2016 10:22
But, you’ll have to figure out the ssh keys, and how to give him access etc. How did I do it for you?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:22
I created the instance
so i had the root access
Manish R Jain
@manishrjain
Apr 02 2016 10:22
I see.
Okay.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:23
I think you gave me the permissions
Manish R Jain
@manishrjain
Apr 02 2016 10:23
Yup. Maybe I should do the same for Pawan as well
Pawan Rawal
@pawanrawal
Apr 02 2016 10:24
i can see the numCpu was definitely a problem, it hasnt crashed for 5 mins now
Manish R Jain
@manishrjain
Apr 02 2016 10:25
Can you share the full command?
Pawan Rawal
@pawanrawal
Apr 02 2016 10:25
./uidassigner --numInstances 1 --instanceIdx 0 --numCpu 1 --rdfgzips $BENCHMARK_REPO/data/rdf-films.gz,$BENCHMARK_REPO/data/names.gz --uids ~/opensource/dgraph/uids/u0
Manish R Jain
@manishrjain
Apr 02 2016 10:25
Also set the memory flag.
You only have 4 Gigs, so set this to 1.5
Because, we currently use more than whatever you set.
You can stop the process, set the flag and rerun
it should be alright.
Pawan Rawal
@pawanrawal
Apr 02 2016 10:27
okay set --stw_ram_mb 1500
Manish R Jain
@manishrjain
Apr 02 2016 10:27
yup
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:28
Actually in RAFT, once there is a leader, all the requests should go through it I guess
all the write requests I mean
Manish R Jain
@manishrjain
Apr 02 2016 10:28
True, but you can still get the request at the worker, and then redirect to master.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:29
yeah
I think itll be better to maintain a member list
lets discuss it sometime
Manish R Jain
@manishrjain
Apr 02 2016 10:30
You mean via gossip?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:31
Maybe gossip, as it would be lightweight
Manish R Jain
@manishrjain
Apr 02 2016 10:31
@pawanrawal Gave you access to GCE. Feel free to create an instance like what Ashwin has.
Would be faster to run things there than laptop.
Gossip still wouldn’t know about the master. So, say you passed in an IP address to a new node.
The message reaches this node. It’s not the master. It can just reply back saying, here’s the master IP address.
sort of like a redirect.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:32
Thats true, but for the master to know the new node that needs to join, we need its information
Manish R Jain
@manishrjain
Apr 02 2016 10:32
And then the new node can try connecting to the master.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:33
It can be done in many ways, have to come up with the best one for us
Manish R Jain
@manishrjain
Apr 02 2016 10:33
Do you have other ideas?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:35
As you mentioned, one way is to forward the request to master. Here, the details of the node that wants to join has to be forwarded too
Manish R Jain
@manishrjain
Apr 02 2016 10:35
We can have a chat tomorrow, if you want.
Instead of forwarding, you can just return the master IP address to the new node.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:35
or we can reply with the masters IP an dthe node can talk to master directly
Yeah
But in case the master fails, none of the followers will have info about other followers
so re election will be a problem
Manish R Jain
@manishrjain
Apr 02 2016 10:36
I don’t follow
The basic logic of master re-election and the peer list still works.
Joining a raft cluster is a separate step.
Once joined, you should get back a list of peers, don’t you?
Othewise, how will the cluster work...
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:38
We somehow have to make sure the everynode is connected to every other node while the node joins the cluster
yeah thats independent of RAFT, we have to handle that
Manish R Jain
@manishrjain
Apr 02 2016 10:39
How does a RAFT cluster talk to each other today? Assuming no new joins...
Surely, they’re talking to each other.. Otherwise, master election would be impossible.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:40
all the followers talk to the master
and master to followers
I havent handled the master failure yet in the experiment code
Manish R Jain
@manishrjain
Apr 02 2016 10:42
You should do that first, before doing the join.
That’s the most important part of RAFT.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:43
Yeah basically returning a peer list when a join happens and establishing connections
Manish R Jain
@manishrjain
Apr 02 2016 10:44
Yeah, something like that.
Pawan Rawal
@pawanrawal
Apr 02 2016 10:45
thanks @manishrjain , will set an instance up
Pawan Rawal
@pawanrawal
Apr 02 2016 10:55
@manishrjain what instance type would be good ?
n1-standard-1 or n1-standard-2
Manish R Jain
@manishrjain
Apr 02 2016 10:55
just use what aswin is using
Pawan Rawal
@pawanrawal
Apr 02 2016 10:56
okay
Ashwin Ramesh
@ashwin95r
Apr 02 2016 10:56
I'm using n1-standard-2
with 30G disk
Pawan Rawal
@pawanrawal
Apr 02 2016 10:57
yup I can see that
Manish R Jain
@manishrjain
Apr 02 2016 11:03
Alright. There you have it: https://discuss.dgraph.io/
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:04
DNS is not yet ready i guess
Manish R Jain
@manishrjain
Apr 02 2016 11:04
works for me.
Maybe not yet propagated to India?
What do you get?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:05
This site can’t be reached

discuss.dgraph.io’s server DNS address could not be found.
DNS_PROBE_FINISHED_NXDOMAIN
Yeah takes some time to propagate
Manish R Jain
@manishrjain
Apr 02 2016 11:09
128.199.218.168
That’s the IP address
You can access it directly.
Btw, sign in with your @dgraph.io email ids.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:12
I don't have that email id
Manish R Jain
@manishrjain
Apr 02 2016 11:12
lol
emm.. let me see. What id do you want?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:14
ashwin, if thats okay
or else ashwin95r
Manish R Jain
@manishrjain
Apr 02 2016 11:17
Hey @koppula, whenever you get a chance, log into https://discuss.dgraph.io as well using your @dgraph.io email id.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:21
the ip changes to discuss.dgraph.io and gives me the same error!
Manish R Jain
@manishrjain
Apr 02 2016 11:21
hmm.. that’s weird.
@pawanrawal @koppula what do you guys see?
Ashwin Ramesh
@ashwin95r
Apr 02 2016 11:21
lol yeah
Prashanth Koppula
@koppula
Apr 02 2016 11:43
@manishrjain
Will do
Prashanth Koppula
@koppula
Apr 02 2016 11:50
I signed up. Waiting for the activation email
Manish R Jain
@manishrjain
Apr 02 2016 11:51
:thumbsup:
Pawan Rawal
@pawanrawal
Apr 02 2016 12:03
Can see board
Prashanth Koppula
@koppula
Apr 02 2016 12:42
Yeah me too. @manishrjain so we're hosting discuss ourselves?
Manish R Jain
@manishrjain
Apr 02 2016 18:01
Yup. On digital ocean.
Ashwin Ramesh
@ashwin95r
Apr 02 2016 19:16
Hey @manishrjain , I pushed the RAFT code with the peerList support in the testing branch of experiments
Have a look at it if free
Manish R Jain
@manishrjain
Apr 02 2016 22:08
Sure. Will do.