Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Mar 25 18:19

    anidotnet on master

    3.5.0-SNAPSHOT (compare)

  • Mar 25 18:18

    anidotnet on gh-pages

    3.4.1 release (compare)

  • Mar 25 18:12

    anidotnet on v3.4.1

    (compare)

  • Mar 25 16:56
    anidotnet commented #212
  • Mar 25 16:56

    anidotnet on master

    3.4.1 release (compare)

  • Mar 25 16:55
    anidotnet closed #212
  • Mar 25 16:55

    anidotnet on master

    fixed #212 (compare)

  • Mar 25 16:54
    anidotnet milestoned #212
  • Mar 25 16:53
    anidotnet labeled #212
  • Mar 25 16:53
    anidotnet assigned #212
  • Mar 25 16:53
    anidotnet reopened #212
  • Mar 25 16:53
    anidotnet commented #212
  • Mar 25 15:03
    angelix commented #212
  • Mar 25 14:54
    angelix commented #212
  • Mar 25 14:53
    angelix commented #212
  • Mar 25 14:49
    anidotnet commented #212
  • Mar 25 14:34
    angelix commented #212
  • Mar 25 14:33
    angelix commented #212
  • Mar 25 11:07
    anidotnet closed #212
  • Mar 25 11:07
    anidotnet commented #212
Anindya Chatterjee
@anidotnet
a. If you want to maintain 2 types of db, you must create 2 separate Nitrite variables.
b. In your closeConnection method, you are closing the db. In-memory db is not persistent. As soon as you close the db, it looses data unlike file store db.
Dinesh
@dinbtechit
Thanks Anindya. Appreciate your response. Actually I just want to maintain the in-memory version of the db. The github code that I provided was more of an example to demonstrate 2 different behaviors. Unfortunately, I see the same behavior even if I don't close the connection and/or use a different variables.
Anindya Chatterjee
@anidotnet
@dinbtechit here is the final version of DatabaseConnection file which works

public final class DatabaseConnection {
    private static final DatabaseConnection DB_CONN = new DatabaseConnection();

    private Nitrite db;
    private ObjectRepository<Employee> repository;

    private DatabaseConnection(){
        db = Nitrite.builder().openOrCreate();
        repository = db.getRepository(Employee.class);
    }

    public static DatabaseConnection getInstance() {
        return DB_CONN;
    }

    /**
     * Inserting into inmemory
     * @param employee
     * @return
     */
    public List<Employee> insertIntoMemory(Employee employee) {
        return insert(employee);
    }

    /**
     * Find from memory
     * @param offset
     * @param size
     * @return
     */
    public List<Employee> findFromMemory(int offset, int size){
        return find(offset, size);
    }

    private List<Employee> insert(Employee employee){
        List<Employee> employees = new ArrayList<>();
        WriteResult wr = null;
        try {
            wr = repository.insert(employee);
            System.out.println("\n Affected Rows : " + wr.getAffectedCount());
            return repository.find().toList();

        } catch (Exception e) {
            System.out.println("Unable to insert transaction. ");
            e.printStackTrace();

        }
        return employees;
    }


    private List<Employee> find(int offset, int size) {
        List<Employee> employees = new ArrayList<>();
        try {
            employees = repository.find(limit(offset, size)).toList();
        } catch (Exception e) {
            System.out.println("Unable to find transactions. ");
            e.printStackTrace();
        }
        return employees;
    }

}
And here is the result
HTTP/1.1 200 OK
Date: Thu, 05 Dec 2019 18:30:39 GMT
Content-Type: application/json
Content-Length: 365
Server: Jetty(9.4.22.v20191022)

[
  {
    "empId": "f843e184-46f0-4967-b3e1-fecbb4dfcf23",
    "address": "From Memory. 123, North Pole."
  },
  {
    "empId": "6a75a311-5cb7-44fd-9ca8-1b7a86ea3b22",
    "address": "From Memory. 123, North Pole."
  },
  {
    "empId": "c2e3c339-26b3-4bb9-8823-41bc94e90065",
    "address": "From Memory. 123, North Pole."
  },
  {
    "empId": "1d40b65a-4ef7-4acf-973e-2b327f591629",
    "address": "From Memory. 123, North Pole."
  }
]

Response code: 200 (OK); Time: 35ms; Content length: 365 bytes
Not only closing the store after every call, you are also calling Nitrite.builder().openOrCreate()before every insert call, which is actually resetting the memory store before every insert.
Dinesh
@dinbtechit
Ahh! that totally makes sense. Massively appreciate that.
Jon Buck
@jonbuck16

Hi, I am not sure I understand the find API for a specific use case.. say I have some JSON as follows...

"tags": [
            {
                "type": "example"
            }
        ],

How do I write a find query that returns documents that have a tag where the key is type or documents that have a type field with the value example?

Anindya Chatterjee
@anidotnet
You have to use elemMatch filter here.
public void testCollectionField() {
        Document document = createDocument("name", "John")
            .put("tags", new Document[] {
                createDocument("type", "example").put("other", "value"),
                createDocument("type", "another-example").put("other", "some-other-value")
            });

        NitriteCollection example = db.getCollection("example");
        example.insert(document);

        document = createDocument("name", "Jane")
            .put("tags", new Document[] {
                createDocument("type", "example2").put("other", "value2"),
                createDocument("type", "another-example2").put("other", "some-other-value2")
            });
        example.insert(document);

        DocumentCursor cursor = example.find(elemMatch("tags", eq("type", "example")));
        for (Document doc : cursor) {
            System.out.println(doc);
        }
    }
nitrite currently does not support indexing on collection field, so for now you have to search un-indexed.
Jon Buck
@jonbuck16
@anidotnet Thanks for the help, will give this a try
Tareq Kirresh
@TareqK
@anidotnet Ive got a crazy idea
About the datagate
Interested ?
The idea is to re-write it as a separate project (as a library), and offer a standalone version as well
This way, we can allow people who want to build apps with the datagate to integrated it better instead of integrating via the database
Especially when it comest to stuff like access rights and access control and sync policies
I'm also thinking we should look at the sync client and use something more real time like a websocket or SSE instead of polling
Making things a bit faster and have higher data consistency
What do you think ?
Anindya Chatterjee
@anidotnet
Your point about refactoring the sync client, I definitely have it in my mind for the next major release 4.x. But I did not understand your proposal about datagate server. What use case do you have in mind? Care to elaborate? As for me, I was planning to rewrite it using golang as a separate application to squeeze maximum throughput during sync. Implementing it as a library I did not have it in my mind unless there is a valid use case.
Tareq Kirresh
@TareqK
The use case is simple : I, as someone who wants to build a mobile application that uses datagate, need integrated authentication and access rights (Ie, I can only see and edit my authorized data), need multiple login tokens (from my different clients), may need to modify data from web/desktop clients directly to the database, etc. In most applications, I'll also need a higher level of consistency for some entities, like invoice serials and stock numbers , etc. So a lot of the time, I'll be running the datagate alongside my application server, and getting data from both. Right now, this means that I'll be integrating via the database and running 2 servers. If the datagate were a library with its own endpoints integrated into my application, this wouldn't be the case, and I have more levels of consistency possible.
Anindya Chatterjee
@anidotnet
@TareqK in its current form you can do the same with datagate application. It's a spring boot application, so you should be good to integrate datagate into your application, provided your application is also a spring boot application. You just have to delete the application file NitriteDataGate.java from datagate.
Tareq Kirresh
@TareqK
I use JavaEE, since I need some of the features. Which is why I want it to be a library, so it's framework agnostic .
Anindya Chatterjee
@anidotnet
How are you planning to build a server component which is framework agnostic? You have to use some kind of framework to expose the endpoints. Moreover, what you are proposing is directly against microservice architecture. You can modify datagate source to fit your access control, but I'll suggest separate your application logic from datagate.
Tareq Kirresh
@TareqK
It can be kept agnostic by writing it as a plain servlet or websocket and using it in any servlet compatible environment. I mean we are going to rewrite it using websockets anyway, which means everything we use will be in the javax.ws package which is compatible with a lot of frameworks of varying complexity (spring , jersey, rest easy, spark, etc).
And as for the Microservices paradigm, it shouldn't be forced on users, they should be able to choose anyway.
Tareq Kirresh
@TareqK
I can whip up an architecture diagram and some design specs If you'd like to highlight how this would work(as code and deployment) and how it can be structured
Anindya Chatterjee
@anidotnet
Ok. Sounds nice. Let me know once you come up with some architecture diagram and some high level design.
Tareq Kirresh
@TareqK
nitrite-cluster.png
nitrite-single.png
Here are the setups for a clustered and for a single instance setup mode
cluster setup.png
Here is the Sequence diagram for how messages will be transferred across multiple clusters to reach all clients subscribed to a collection
Tareq Kirresh
@TareqK
This kind of setup, whether single mode or listening to a kafka cluster, is implemented using websockets from the javax.ws package and json-rpc, meaning that it supports all server environments and frameworks that support it. We can provide the websocket class, with callbacks for authentication, session management, authorization, etc, for people to override as they see fit(with of course, a default implementation for us). These things can be passed as parameters in a configuration object or the web.xml(or servlet holder), allowing easy configuration via text or code. I personally believe that doing this, we can get rid of a lot of performance issues(polling is expensive on the client and server side), and increase overall service quality. Additionally, we should publish the json-rpc API, allowing anyone to create a datagate client the way they see fit, including via javascript for a web application (a la pouch db) or for other kinds of devices(IOT comes to mind), allowing a more extensible ecosystem based on nitrite
I know that nitrite is designed to be a simple database, but nitrite datagate, i think , should be something that can be used to build scalable applications
Of course, we will provide default implementations for everything, which may or may not suit people.
@anidotnet what do you think?
Anindya Chatterjee
@anidotnet
I'll strongly suggest before start implementing it, please come-up with a proper protocol. Currently, I am working on rewriting nitrite for making it more modular and extendable. You can use your own store, indexers, replication logic etc. I have also started working on writing nitrite for typescript. I have also thought about rewriting datagate to make it compatible with different platform. It looks like your idea is on the similar line. Only I have not thought about making datagate as a library.
Anyway if you want, first start with the replication protocol which should be language and platform agnostic, performant and scalable. For reference, you can take a look at the couchbase replication protocol - https://blog.couchbase.com/data-replication-couchbase-mobile/
Tareq Kirresh
@TareqK
Will do! But the protocol itself is second to the architecture it runs on IMHO. The protocol and message format itself can be changed later on in production, but bad arch is bad arch. Anyways, I take it that you are warmed up to the idea of this more customizable implementation?
Also is JBus still a thing? Or did you stop maintaining it altogether ?
Anindya Chatterjee
@anidotnet
Please go ahead with datagate. I may not invest significant time on it as I am busy with rewriting nitrite. But surely we will sync up when you have solid arch and protocol and when I need to rewrite the replication client module.
JBus is still supported, just not actively developed. Do you have anything on your mind regarding it?
Tareq Kirresh
@TareqK
Not much, it's just that I have been getting into Domain Driven Design lately and need a simple message bus to do some testing(I don't want to host Kafka or RabbitMQ). If JBus still works, I might use it
Tareq Kirresh
@TareqK
@anidotnet which branch should I fork to get started on?
Anindya Chatterjee
@anidotnet
I'll suggest you create a new repository altogether for datagate. Anyway we are not going to use current implementation in the next major release after nitrite rewrite. Once it is complete, you can move the repo to the org.
Anindya Chatterjee
@anidotnet
@TareqK and do not start anything on client yet. Once server design and implementation is complete, we will talk about that, meantime nitrite rewrite will be ready.
Tareq Kirresh
@TareqK
My thoughts exactly.