by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 24 20:09
    anidotnet commented #216
  • Jun 24 20:03
    frett27 added as member
  • Jun 24 14:35
    anidotnet commented #228
  • Jun 24 09:09
    sheinbergon commented #228
  • Jun 23 18:04
    aleksandar-stefanovic commented #218
  • Jun 23 17:45
    aleksandar-stefanovic commented #218
  • Jun 23 09:56
    anidotnet commented #228
  • Jun 23 09:55
    anidotnet commented #228
  • Jun 23 09:21
    sheinbergon commented #228
  • Jun 23 09:19
    sheinbergon opened #228
  • Jun 22 15:44
    anidotnet commented #216
  • Jun 22 11:12
    frett27 commented #216
  • Jun 21 19:03
    anidotnet commented #216
  • Jun 21 18:31
    frett27 commented #216
  • Jun 19 10:22
    anidotnet commented #216
  • Jun 19 08:56
    frett27 commented #216
  • Jun 16 12:14
    anidotnet closed #227
  • Jun 16 12:14
    anidotnet commented #227
  • Jun 16 12:06
    aukhatov opened #227
  • Jun 10 12:07
    aukhatov commented #226
and here is the repository
I still need to update some stuff and add actual database implementations and actual authentication implementations, but I dont want to continue implementation before we agree on the actual API. If you feel like there are enough features to begin with, ill get started on finishing the implementations, and we can sync on the client soon after.
Anindya Chatterjee
@anidotnet
Thanks @TareqK . I'll take a look at it whenever I get some time. Meantime, provide an example on how to spin a spring boot server using you library. I don't want to desert current spring boot datagate consumers also.
Tareq Kirresh
@TareqK
@anidotnet Ill get on it tonight, but I've already spun up a tomcat one that runs like the spring boot one
Though I may not use thymeleaf for the UI stuff (I'll see about it later )
Anindya Chatterjee
@anidotnet
It's better not to use thymeleaf to handle websocket call.. Instead use vuejs or angular for the UI
Tareq Kirresh
@TareqK
Well we shouldnt worry about UI for now
I just finished the mongodb(default) implementation
works fine.
Anindya Chatterjee
@anidotnet
great
Tareq Kirresh
@TareqK
Now im thinking that the default authenticator(that ships with the library) always authenticates
while the standalone version actually does real authentication
what do you think?
I want to do this because while most people have an authentication system they want to tie into for their application, very few will actually want to write their implementation of change lists
and one more issue im thinking of
What about deletion??
is deletion allowed?
Anindya Chatterjee
@anidotnet
IMO authentication should not be part of the library. Many people want to use their own security mechanism in their server software. What do you mean by deletion allowed? It is sync operation,. Deletion in local collection must be propagated to their replicas via datagate.
Tareq Kirresh
@TareqK
Which leads me to ask how updates work. Do we send differenitials or the whole entity?
document* not entity
Tareq Kirresh
@TareqK
because for the sake of simplicity, differentials might not be viable ATM
Tareq Kirresh
@TareqK
Okay, so ive gone ahead and finished the implementation of the standalone server(with authentication via mongodb)
What is left is to add the mongodb and admin user configuration, similar to the old datagate ones, and we can get started on writing client stuff
Anindya Chatterjee
@anidotnet
When I get sometime over this weekend, I'll look into your project and we will discuss on our next course of action. Meantime please add a spring-boot implementation into the project.
Tareq Kirresh
@TareqK
HELOoooOOOOoooOOOoooOOOooOOOooOOOOooO Bois and Grills
Datagate Spring boot project is up
so is a Spring boot library you can import with an @Import in your main class
I decided to reuse the Jsr356 implementation for spring boot to cut down on written code
@anidotnet Please tell me when you see the latest changes
Anindya Chatterjee
@anidotnet
@TareqK I have got some time to see your changes. Here are my concerns
  1. you should remove authentication from library and delegate it to the implmentor. Implementor can choose from basic auth, oauth or any other custom mechanism. Instead in library add some mechanism to add some auth token in every request and delgate the validation of the token to the implementor.
  2. you should remove factory classes from nitrite-datagate module, make it more like a api library with only message handler and supporting classes. Default implementations you can provide in nitrite-datagate-javaee module and people can wire it at their own will. spring boot user can create beans and wire them via spring's autowiring mechanism. No need for factory classes.
  3. you should remove datagate user service from library.
  4. you add validation that a user has access to the collection passed in request or not
  5. for replication, you should maintain one mongo collection for each user@nitrite-collection. separate each user's data in different mongo collection.
  6. the core sync logic is not complete I guess. My suggestion is to keep the core sync logic same as old version. Add websocket interface in place of rest and add pagination in every request and response message. Make it configurable so that client can choose to send/recieve n number of changes at a single go.
  7. there is no locking mechanism in your sync logic. what happens when 2 clients of same user trying to update the same collection? locking logic is there in the old version, please use that.
  8. for datagatebus you can use jbus or any other message bus, but in datagate library keep the interface only. in jee or spring implementation provide actual implementation.
and lastly, as I said earlier I am likely to provide a golang based full datagate server implementation. I might not prefer java based implementation (but I have not decided yet). So be prepare to support your library in long run :D
Tareq Kirresh
@TareqK
About #5 and #1 , websockets don't support authentication via request, so that's not really possible (so I might be able to add some token authentication, but the user needs to send it after opening), and check again, the collections are in the specific format of user@collection
The rest I will work on , I just didn't want to invest too much time on something I wasn't sure is needed
Also, I'm keeping the factories in the base package, because I use them to switch what implementation is returned
I'm not quite sure about pagination because that's REST thinking, a socket isba streaming API so it feels kind of strange
And yes I want to maintain it anyhow, because it could be used for general purpose sync, not just nitrite sync
Tareq Kirresh
@TareqK
Have we agreed on the API though ? Did you look at the spec @anidotnet
Tareq Kirresh
@TareqK
Guys does anyone want to help me write the syncing spec ?
Anindya Chatterjee
@anidotnet
Hi @TareqK my bad on #5. It was ok, I missed it somehow first time. I'll still strongly suggest to remove factories from core library and put it inside the implementation library as it makes more sense to use factory classes to create implementation classes. By pagination I actually meant for throttling. We should not send all changes in one message. If the document size is large, it will create network bottleneck. We should keep each message small and quick. So before sending actual changes, we should handshake throttling info with the server so that server and client understands how much size they can mutually handle per message.
I am writing the sync spec draft, I'll share withing a day or two
Tareq Kirresh
@TareqK
But we agree on using Json-rpc over websocket for messages yes?
Oh and throttrling makes sense this way yeah , I can see the business case
Tareq Kirresh
@TareqK
@anidotnet Ive refactored the projects into an api and an impl project. The api project still has some factories, but those are only there so that i dont have 500000 places where i call reflection code. Ive also added a cleaner way for users to provide their configs, similar to an Application.setApplication() call in JavaEE, and added it to the correct projects(this has the added benefit of users being able to hide their own implementations from inside their projects, that is, have seperate datagate implementation jars from their main jars and calling them as dependencies, which is useful in a microservices-oriented organization). Hopefully this is a structure we can agree on. I havent added pagination and all the other stuff - yet - but i intend to soon. When you have the time, open a few github issues for me with the exact description of what we need to do, since I assume after writing the draft you would have added/removed some requirements.
Anindya Chatterjee
@anidotnet
@/all does anyone has any experience with CRDT? Planning to use it for nitrite replication, any help would be appreciated..
Tareq Kirresh
@TareqK
Interesting topic. What do you have in mind ? Also, what should the server do for CRDT ? Or have you not determined that yet ?
Tareq Kirresh
@TareqK
Ive been reading into CRDT(loving it so far), but the way I see it, the whole setup of the current datagate server needs to change; I would need to track changes rather than current state(ie, work in an event sourcing manner) and potentially applying changes for a pre-calculated state(for accelerated retrieval). I think this sounds fantastic, but, now that we have this in mind, how about instead of a nitrite-only sync server, we put a general purpouse sync server/client spec and implement that instead?
Anindya Chatterjee
@anidotnet
I am also thinking in the same direction. a general sync server spec over websocket and then we will think about integrating in nitrite
but before that I need to dig down more on data structure requirement and see what kind of ds changes I need to make in nitrite. How to manage the whole MVMap as a CRDT is a task I am currently concentrating.