Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 10:33
    Gillani0 commented #209
  • Feb 27 18:28
    anidotnet closed #209
  • Feb 27 18:28
    anidotnet commented #209
  • Feb 27 13:17
    Gillani0 opened #209
  • Feb 27 13:10
    Gillani0 closed #206
  • Feb 25 08:05
    anidotnet commented #207
  • Feb 25 08:02

    anidotnet on master

    added a test with external nitr… (compare)

  • Feb 25 07:44
    anidotnet closed #207
  • Feb 25 07:44
    anidotnet commented #207
  • Feb 25 07:38
    anidotnet closed #208
  • Feb 25 07:38
    anidotnet commented #208
  • Feb 24 22:19
    Double-O-Seven opened #208
  • Feb 24 22:17
    Double-O-Seven opened #207
  • Feb 24 07:38
    anidotnet commented #206
  • Feb 24 07:36
    anidotnet commented #206
  • Feb 24 07:27
    Gillani0 commented #206
  • Feb 24 06:55
    anidotnet commented #206
  • Feb 24 06:51
    Gillani0 opened #206
  • Feb 18 06:51
    anidotnet closed #204
  • Feb 18 06:51
    anidotnet commented #204
Tareq Kirresh
@TareqK
Oh and throttrling makes sense this way yeah , I can see the business case
Tareq Kirresh
@TareqK
@anidotnet Ive refactored the projects into an api and an impl project. The api project still has some factories, but those are only there so that i dont have 500000 places where i call reflection code. Ive also added a cleaner way for users to provide their configs, similar to an Application.setApplication() call in JavaEE, and added it to the correct projects(this has the added benefit of users being able to hide their own implementations from inside their projects, that is, have seperate datagate implementation jars from their main jars and calling them as dependencies, which is useful in a microservices-oriented organization). Hopefully this is a structure we can agree on. I havent added pagination and all the other stuff - yet - but i intend to soon. When you have the time, open a few github issues for me with the exact description of what we need to do, since I assume after writing the draft you would have added/removed some requirements.
Anindya Chatterjee
@anidotnet
@/all does anyone has any experience with CRDT? Planning to use it for nitrite replication, any help would be appreciated..
Tareq Kirresh
@TareqK
Interesting topic. What do you have in mind ? Also, what should the server do for CRDT ? Or have you not determined that yet ?
Tareq Kirresh
@TareqK
Ive been reading into CRDT(loving it so far), but the way I see it, the whole setup of the current datagate server needs to change; I would need to track changes rather than current state(ie, work in an event sourcing manner) and potentially applying changes for a pre-calculated state(for accelerated retrieval). I think this sounds fantastic, but, now that we have this in mind, how about instead of a nitrite-only sync server, we put a general purpouse sync server/client spec and implement that instead?
Anindya Chatterjee
@anidotnet
I am also thinking in the same direction. a general sync server spec over websocket and then we will think about integrating in nitrite
but before that I need to dig down more on data structure requirement and see what kind of ds changes I need to make in nitrite. How to manage the whole MVMap as a CRDT is a task I am currently concentrating.
Tareq Kirresh
@TareqK
How about we do separation of concerns and spec into CDRT only (client-serer) and we'll see how to apply
Like maybe a tombstone prefix postfix, operation specifiers,etc
And it will come on on it's own I'm sure
Anindya Chatterjee
@anidotnet
What do you have in your mind? Care to write a small POC?
Tareq Kirresh
@TareqK
Im not quite sure,
What i do have in mind right now is to do the simplest viable CRDT implementation
So i want collection add/remove
and number increment/decrement
and string overwrites
what remains to be seen is how we will handle actually removing properties
I havent seen anything in CRDT concerning property deletion
Tareq Kirresh
@TareqK
I'm reading some academic papers on CRDT, trying to wrap my head around all the math and proofs behind it, I'll probably finish them tonight, and I'll be able to have a clear Id of what I want to do
Tareq Kirresh
@TareqK
@anidotnet @all https://www.youtube.com/watch?v=vBU70EjwGfw i suggest everyone intrested in CRDTs watch this quick intro, its super useful and practical. Also, i think we need a seperate channel for CRDTs
@anidotnet Also, i have a private wiki where i am keeping notes(I use it for my master's projects and other thigns), and if you are interested with reading them, DM me so i can give you access
Anindya Chatterjee
@anidotnet
@TareqK about handling property deletion, I was thinking how about we use ORSet crdt and track only the operations on a NitriteMap (ADD, UPDATE, REMOVE operations). We already store last modified timestamp (since epoch) in document. We can use that or we can use VectorClock and tag it via NitriteId only. So in essence, we track operations per NitriteId and timestamp stored in documents.
Let me know what do you think if it is a viable solution
Tareq Kirresh
@TareqK
We still need to add a UUID to the documents and fields, aside from the nitrite ID, so that operations on deleted / set documents don't communtate on new ones. We will probably need a metadata collection that maps every field and object to a UUID. I am not sure how to go about this, maybe each collection will have a <collection>_metadata collection, tracking all the UUIDs for all the Fields. While it's good that causality is more or less solved ( we only need the Vector clock), we still have the problems with nodes that drop out.
Also, we need to discuss Ahead of Time compiled entities server side, and how we are going to store change operations. I imagine they can be stored in a database, or we can store them in a MQ like Kafka. But in all cases, this needs to be stored for late nodes and traceability.
I still haven't finished reading all the academic mumbo jumbo behind CRDTs though
And the problem of metadata storage is fairly large
Anindya Chatterjee
@anidotnet
Don't go into meta data route. It is not practically viable. Academic algorithms are good for theories but in practice it is very hard to implement and maintain. Meta data route will exponentially demand spaces to store all data.
Instead what I outlined, I found some similar implementation on github today. Here is the link - https://github.com/cyber-front/json-crdt
go through the code, idea is somewhat similar, keep track of operations and they are using JsonPatch to merge the field changes. Fairly simple and not much resource hogging
take a look and let me know if we can start on the basis of this project
Anindya Chatterjee
@anidotnet
@TareqK ^^
Tareq Kirresh
@TareqK
So it's not full CRDT, we are just using the commutative operations ?
Tareq Kirresh
@TareqK
@anidotnet Ive looked a little into it, and im not quite sure. Ill need a bit more time over the weekend
Anindya Chatterjee
@anidotnet
Sure @TareqK .. no hurry.. take all the time we need. I'll also play with it in nitrite to see how it goes.
Tareq Kirresh
@TareqK
@anidotnet hi I'm just updating you real quick. Haven't had much time this week, I have been super busy with clients + job + master's exams, but starting Saturday I can finally get around to this project. Have you made any progress ?
Anindya Chatterjee
@anidotnet
I have worked on the replication library and created a basic communication over websocket
I'll give you access to the private repo, so that you can see what I have done so far, in test code there is a mock server also.. there are couple of issues still in the code, working on it.
Anindya Chatterjee
@anidotnet
I have given you access to the private repo. please remember not to push any changes to this repo.
Tareq Kirresh
@TareqK
@anidotnet can I branch and do PRs?
Anindya Chatterjee
@anidotnet
you can but datagate server is not in this repo. this repo is refactored nitrite with replication module. There are significant api changes also. have a look - https://github.com/anidotnet/nitrite-java
after little bit of more testing with replication module, I'll start working on golang datagate server
Angelos Veglektsis
@angelix
Hi all, based on the documentation if indexed, it’s java type should be primitive or implement java.lang.Comparable.
This means that i cannot have NULL as a document value at all?
Tareq Kirresh
@TareqK
@anidotnet Im finally back in comission
what did I miss while i was gone
Anindya Chatterjee
@anidotnet
@TareqK new replication module is ready over websocket. Here is the link - https://github.com/anidotnet/nitrite-java
And started working on golang datagate server - https://github.com/anidotnet/nitrite-datagate
And started working on golang datagate server - https://github.com/anidotnet/nitrite-datagate
Anindya Chatterjee
@anidotnet
Here is the draft sequence diagram of how things will work - https://www.websequencediagrams.com/cgi-bin/cdraw?lz=dGl0bGUgRGF0YUdhdGUgTWVzc2FnZSBDb21tdW5pY2F0aW9uCgpSZXBsaWNhIChSMSktPlNlcnZlcjogQ29ubmVjdAAtCApub3RlIGxlZnQgb2YgAB0IVmFsaWRhdGVzIEF1dGggdG9rZW4gaW4gbQAuBwAwBm92ZXIgAFoMCiAgICBTb21laG93IG5lZWQgdG8gYmEAGwpmcm9tIAAhBXNlbmRpbmcAgTwJRmVlZABXCCB0bwBEBgCBLgUgYmVmb3JlIHJlY2VpdmluZwCBOwhBY2sKZW5kIG5vdGUKCmFsdACBMQYAgSgGADYLAIFyClN0b3JlcyByAIISB2lkABkNAIIkDACCHwlBY2sAgiIJICAgAGgFAIImCQAlDQCBeAUgICAAggkJY2FuIG5vdwCBMgcAgyYKRmVlAHMGAIE0CWVsc2UAgmwIaW9uIGZhaWwAHQcAgQcWRXJyb3IAcScAgmoGICAgIENsb3NlcyBjAIN8BmlvbiB0byBzAIQTBQCBNAlhbmQgd2lsbCBub3QAgTEJYW55IGZ1cnRoAB4LYwCEYQxzAINUBgA9D3VudGlsIG5leHQgc3VjY2Vzc2Z1bACEeQgAgXMPbmQAhSEQAIMBDkZpbmQgbGFzdCBzeW5jIHRpbWUADiJjaGFuZ2VzIHNpbmNlADQKAIYDGEJhdGNoQwAvBVN0YXIAhhUPcmlnaACDZREAhW0FSW4AhEoJa2VlcCB0aGUgbGlzAIZNBU5pdHJpdGVJZHMgb2YAhhsFdGhlIERvY3VtZW50cwCFfwkAgwEJcy4Ahn8HAIZIBXdvdWxkAIYoBSBiYWNrAIEXDEFjawCGKAl3aXRoAIZ4BQBkE2FjY2VwdGVkIGJ5AIdZBy4Ahy0IAF0GaWxsIHJlbW92ZSB0aG9zZSBpZACDSQdpdCdzAIFFBS4gRmluYWxseQCHXQVhZnRlciBleACCWQYgaXMgY29tcGxldGVkIHRoaQAtBiBzaXplIHNob3VsAIZhBmJlIDAuAIcmCwCCOxoAhSwHSWYAgzgJZXhpcwCCJgdzAINNCQCIaQVpbiBjaHVua3MsAGsGaXMgc2V0cyBpAIMLCQCKCAUAhSwGbmZpZwCIKAsAh2MWAIQBBUFjawCKBhQAiVwFAIJ1DwCGKQVjb250YWluAINvCACKBAUAgmYeAIkwCwCEbSFDb250aW51AIthCQCBBzMAiRkGAIpKCHJvYWRjYXN0AIUmBQBVFACFJAZvIGEAgVYGbmVjdGVkIHBlZXJzAF1cAIJ8FwCKLQ0AjUEWRmVlZAA9BQCENiFXaGVuIHRoZXIAhTUFbm8gbW9yZQCIIgkAig0FbgCMEwYAhnIFAIgKC0UAihcFdGgAiHgQAIMfK0VuZACIMyMAiy0HAGwPAIdjCgCEZgsAjUUHSWQsAIZfBmF0Y2hTaXplAItEBQCBEBgAhV4bRW4AgkIFAIpvHENoZWNrAI9FBmpvdXJuYWwgZm9yAI0uByBlbnRyaWVzAJBnF1JldHJ5ABwPIGluAI4ODgCOexYAg1kJAIccGwCLKRQAi1EbAIYIJACJPAYgZW5zdXJlAIYLBQCRHghzZW50IHRvAJBHCACJXwZ5AIt-BQCLTQVyYWNrIG9mAIMZCgCOWAVuAIwJCCBtYXAAiGcmAIduFwCBOh8AggYhAIUADACOXxxTYXZlAJMSCWhlYWRlcgCOewUgYXMAjnQeAI5LDQCEUgcAi1scAJQ6BUZyb20AkjIFb24AlFMJAJEMBXN0b3JlIHRoAJMCBnRpbWVzdGFtcCBvZgCRGwUAgmgFciBmAJQ9CwCNJwYAgRQRAIpZIACEYSsAgWJVAIYtQQCGOTYAhkEvAIppKwCEFkcAgxcZaXMAlUwHAI4PEQCIbQptb3YAl2wOAIlFHQCWGRAKCgo&s=rose