by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 24 20:09
    anidotnet commented #216
  • Jun 24 20:03
    frett27 added as member
  • Jun 24 14:35
    anidotnet commented #228
  • Jun 24 09:09
    sheinbergon commented #228
  • Jun 23 18:04
    aleksandar-stefanovic commented #218
  • Jun 23 17:45
    aleksandar-stefanovic commented #218
  • Jun 23 09:56
    anidotnet commented #228
  • Jun 23 09:55
    anidotnet commented #228
  • Jun 23 09:21
    sheinbergon commented #228
  • Jun 23 09:19
    sheinbergon opened #228
  • Jun 22 15:44
    anidotnet commented #216
  • Jun 22 11:12
    frett27 commented #216
  • Jun 21 19:03
    anidotnet commented #216
  • Jun 21 18:31
    frett27 commented #216
  • Jun 19 10:22
    anidotnet commented #216
  • Jun 19 08:56
    frett27 commented #216
  • Jun 16 12:14
    anidotnet closed #227
  • Jun 16 12:14
    anidotnet commented #227
  • Jun 16 12:06
    aukhatov opened #227
  • Jun 10 12:07
    aukhatov commented #226
Tareq Kirresh
@TareqK
And as for the Microservices paradigm, it shouldn't be forced on users, they should be able to choose anyway.
Tareq Kirresh
@TareqK
I can whip up an architecture diagram and some design specs If you'd like to highlight how this would work(as code and deployment) and how it can be structured
Anindya Chatterjee
@anidotnet
Ok. Sounds nice. Let me know once you come up with some architecture diagram and some high level design.
Tareq Kirresh
@TareqK
nitrite-cluster.png
nitrite-single.png
Here are the setups for a clustered and for a single instance setup mode
cluster setup.png
Here is the Sequence diagram for how messages will be transferred across multiple clusters to reach all clients subscribed to a collection
Tareq Kirresh
@TareqK
This kind of setup, whether single mode or listening to a kafka cluster, is implemented using websockets from the javax.ws package and json-rpc, meaning that it supports all server environments and frameworks that support it. We can provide the websocket class, with callbacks for authentication, session management, authorization, etc, for people to override as they see fit(with of course, a default implementation for us). These things can be passed as parameters in a configuration object or the web.xml(or servlet holder), allowing easy configuration via text or code. I personally believe that doing this, we can get rid of a lot of performance issues(polling is expensive on the client and server side), and increase overall service quality. Additionally, we should publish the json-rpc API, allowing anyone to create a datagate client the way they see fit, including via javascript for a web application (a la pouch db) or for other kinds of devices(IOT comes to mind), allowing a more extensible ecosystem based on nitrite
I know that nitrite is designed to be a simple database, but nitrite datagate, i think , should be something that can be used to build scalable applications
Of course, we will provide default implementations for everything, which may or may not suit people.
@anidotnet what do you think?
Anindya Chatterjee
@anidotnet
I'll strongly suggest before start implementing it, please come-up with a proper protocol. Currently, I am working on rewriting nitrite for making it more modular and extendable. You can use your own store, indexers, replication logic etc. I have also started working on writing nitrite for typescript. I have also thought about rewriting datagate to make it compatible with different platform. It looks like your idea is on the similar line. Only I have not thought about making datagate as a library.
Anyway if you want, first start with the replication protocol which should be language and platform agnostic, performant and scalable. For reference, you can take a look at the couchbase replication protocol - https://blog.couchbase.com/data-replication-couchbase-mobile/
Tareq Kirresh
@TareqK
Will do! But the protocol itself is second to the architecture it runs on IMHO. The protocol and message format itself can be changed later on in production, but bad arch is bad arch. Anyways, I take it that you are warmed up to the idea of this more customizable implementation?
Also is JBus still a thing? Or did you stop maintaining it altogether ?
Anindya Chatterjee
@anidotnet
Please go ahead with datagate. I may not invest significant time on it as I am busy with rewriting nitrite. But surely we will sync up when you have solid arch and protocol and when I need to rewrite the replication client module.
JBus is still supported, just not actively developed. Do you have anything on your mind regarding it?
Tareq Kirresh
@TareqK
Not much, it's just that I have been getting into Domain Driven Design lately and need a simple message bus to do some testing(I don't want to host Kafka or RabbitMQ). If JBus still works, I might use it
1 reply
Tareq Kirresh
@TareqK
@anidotnet which branch should I fork to get started on?
Anindya Chatterjee
@anidotnet
I'll suggest you create a new repository altogether for datagate. Anyway we are not going to use current implementation in the next major release after nitrite rewrite. Once it is complete, you can move the repo to the org.
Anindya Chatterjee
@anidotnet
@TareqK and do not start anything on client yet. Once server design and implementation is complete, we will talk about that, meantime nitrite rewrite will be ready.
Tareq Kirresh
@TareqK
My thoughts exactly.
Can i use maven instead of gradle ? Datagate is server side, and I have a lot of experience with maven
Anindya Chatterjee
@anidotnet
sure
Tareq Kirresh
@TareqK
@anidotnet https://github.com/TareqK/nitrite-datagate Ive started the repo and added the basic structure for the project. For the sake of squeezing out performance, im keeping reflection to a minimun and instead using Interfaces where possible
Tareq Kirresh
@TareqK
@anidotnet Ive finalized the way I imagine the structure will work with dummy implementations. Im going to try it out with a spark application to see how it
"feels" to use in anapplication
Ive made the authenticator and the message bus implementable by the user
this allows one to use any messaging bus they want(if they implement it) and any authentication back end for usernames and passwords they want
Anindya Chatterjee
@anidotnet
ok
Tareq Kirresh
@TareqK
@anidotnet I got the proof of concept of the API out. Ive also commited a standalone version, but i dont have a build for an uber jar yet. docs are here https://github.com/TareqK/nitrite-datagate/blob/master/nitrite-datagate/docs/Nitrite-Datagate-API.md
and here is the repository
I still need to update some stuff and add actual database implementations and actual authentication implementations, but I dont want to continue implementation before we agree on the actual API. If you feel like there are enough features to begin with, ill get started on finishing the implementations, and we can sync on the client soon after.
Anindya Chatterjee
@anidotnet
Thanks @TareqK . I'll take a look at it whenever I get some time. Meantime, provide an example on how to spin a spring boot server using you library. I don't want to desert current spring boot datagate consumers also.
Tareq Kirresh
@TareqK
@anidotnet Ill get on it tonight, but I've already spun up a tomcat one that runs like the spring boot one
Though I may not use thymeleaf for the UI stuff (I'll see about it later )
Anindya Chatterjee
@anidotnet
It's better not to use thymeleaf to handle websocket call.. Instead use vuejs or angular for the UI
Tareq Kirresh
@TareqK
Well we shouldnt worry about UI for now
I just finished the mongodb(default) implementation
works fine.
Anindya Chatterjee
@anidotnet
great
Tareq Kirresh
@TareqK
Now im thinking that the default authenticator(that ships with the library) always authenticates
while the standalone version actually does real authentication
what do you think?
I want to do this because while most people have an authentication system they want to tie into for their application, very few will actually want to write their implementation of change lists
and one more issue im thinking of
What about deletion??
is deletion allowed?
Anindya Chatterjee
@anidotnet
IMO authentication should not be part of the library. Many people want to use their own security mechanism in their server software. What do you mean by deletion allowed? It is sync operation,. Deletion in local collection must be propagated to their replicas via datagate.