pubkey/rxdb

Seek help and do not forget to help others.

snookyly

@snookyly

This message was deleted

@heleg Hi, you can compare with my nodejs typescript sample project here: https://github.com/snookyly/rxdb-filefolder.

Anton Melser

@AntonOfTheWoods

@AntonOfTheWoods We only allow couchdb compatible names so you do not have to rename your collection when you add replication. https://docs.couchdb.org/en/stable/api/database/common.html#put--db

Thanks.

Sanni

@alexSanni

ist there an best practice guidline to setup the folder structure

Phillipe Murray

@bilicio

@snookyly the folder you try to put the db on already exists? "Data" folder? Can you see the database file there?

heleg

@heleg

@heleg Hi, you can compare with my nodejs typescript sample project here: https://github.com/snookyly/rxdb-filefolder.

Thank you, very much! I found the problem. It's not about rxdb, it looks like an electron-forge bug.

snookyly

@snookyly

@bilicio yes. DB file is there, but I'd like to browser data from browser.

Anton Melser

@AntonOfTheWoods

Just to validate, the example for GraphQL subscriptions has
const query = `subscription onHumanChanged { humanChanged { id } }`;

But that could equally well be any old string or anything just to indicate to RxDB that it needs to sync immediately, rather than wait until the next scheduled sync time, right? I ask because I am updating multiple lines and didn't know whether I should provide the first id, last id or even an id at all. Thanks!

Anton Melser

@AntonOfTheWoods

Just to be clear - I have tested putting nonsense in both the query and the response and I am still getting the desired result. However, can I rely on there not being any conformance between the two going forward?

Anton Melser

@AntonOfTheWoods

Actually it appears that maybe it does matter, at least that the field in the query (in my above comment "id") needs to exist in the return type of the subscription query. Is that right?

Daniel Meyer

@pubkey

@AntonOfTheWoods it is just a "ping" for rxdb to let it know that it must sync something. The data you send over the socket is not really relevant.

Anton Melser

@AntonOfTheWoods

@pubkey , perfect, thanks!

Anton Melser

@AntonOfTheWoods

I can change hundreds of documents in a short space of time, which takes quite a while to sync to my GraphQL endpoint. It works, but it takes a few minutes to finish. Is there some way to batch these? The docs suggest there is an optional batchSize parameter to the push query but if the Mutation only takes a single document I'm not sure how you can send batches to it... Or is there some magic that detects if the mutation takes a list? I tried putting the batchSize : 10 but it didn't seem to do anything.

Anton Melser

@AntonOfTheWoods

Further looking at the unit tests I still can't work out whether there are push queries that take multiple documents. It seems that the pull can definitely have multiple values but not push

Paul Cinnamond

@pcinnamond_gitlab

We have old web clients in the field that have local copy of data and would like to completely replace all backend collections that clients sync with (push and pull). When we replace the couchdb (or graphql backend db table), is there a way to tell rxdb clients not to push old client data to the new database? Maybe clients can detect their revision came from another db so it is discarded instead of pushed?

Anton Melser

@AntonOfTheWoods

@pubkey can you confirm that as it stands, not only is there no batching of push to GraphQL but also it is only possible to run serially, meaning there is no possibility of parallelising push calls at all? (method runPush of src/plugins/replication-graphql/index.ts)

Daniel Meyer

@pubkey

@AntonOfTheWoods batches run on pull only. For push, we push single documents in serial. The reason is, that this makes it way easier to handle errors because the client exactly knows which document was processed correctly be the server and can just retry the failing ones. But I see the reason for having bulk pushes, and it is definitely possible to implement them with the assumption that the server handles the whole bulk in an atomic operation.

@pcinnamond_gitlab I always just change the name of the client side database. Then all clients start with an empty one again.

Paul Cinnamond

@pcinnamond_gitlab

Thanks Daniel. I might be missing something but what if lots of PWA client apps are in live sync mode and we don't have access to change the database name on their side? My thought is we have to stop service on the old couchdb and graphql endpoints and then have everyone update their clients but maybe their is a way to configure rxdb to not push on sync if some db id or something changed?

Phillipe Murray

@bilicio

@pcinnamond_gitlab i think you could create a collection with some version variable like var version:1.0 and listen to this variable on the client, if this variable change you execute .remove() of the database .

Daniel Meyer

@pubkey

@pcinnamond_gitlab yes I understand. You likely need custom logic to handle these cases

Paul Cinnamond

@pcinnamond_gitlab

Thanks. Guess rxdb cannot do it all but would be a neat feature. Say for a test team that frequently resets databases, it would be nice to set up a policy at a high level. I will probably go with the .remove() approach when a db replace is detected but key trick is making sure the client variable gets set before any live sync pushes.

Anton Melser

@AntonOfTheWoods

@AntonOfTheWoods batches run on pull only. For push, we push single documents in serial. The reason is, that this makes it way easier to handle errors because the client exactly knows which document was processed correctly be the server and can just retry the failing ones. But I see the reason for having bulk pushes, and it is definitely possible to implement them with the assumption that the server handles the whole bulk in an atomic operation.

Thanks for the clarification @pubkey . I picked that up from the code but wanted to make sure. For my use case I basically must have this, so the option is either to roll my own outside of RxDB or try and have it integrated.

I have three DB uses - pull-only (a very large ref collection), push-pull (normal user collections) and push-only (event log). Initially the event log seemed like a really bad fit for RxDB but with batching and some cleaning it would seem that I could have a single interface to idb, which seems very elegant to me. Because the normal user collections can (rarely but for each user a few times) have up to hundreds of insertions/updates per minute, serial updates mean a seriously degraded experience.

So you would be open to a PR for batching? In order to keep maximum backwards compatibility with a nice migration path, would trying to detect whether the pushQuery takes an object or list be how you see the user API?

Daniel Meyer

@pubkey

Yes I am open for a PR. But we should explicitly have a config parameter to define if the frontends sends data in batches or single documents.

we could then make this the default settings in the next major release

Martin Foerster

@phal0r

Hi @pubkey ,
I have a question regarding hooks for plugins: Is there a hook, that could be run before and after the whole migration process? I only saw hooks for each migrating document.

Keegan

@keggatron_gitlab

Hey guys, currently working on an offline first app at work that uses RxDB with graphQL. Had a question about relations. So say if a user created an entity offline, a contact, then they created a separate entity that would be related to that contact like an animal that the contact owned. What is the best practice for controlling the database entries as to make sure that the contact is added to the online database first and that the animal has the updated contact id associated with it? Any guidance as to what I should be reading or how to do it would be greatly appreciated.

Daniel Meyer

@pubkey

@phal0r no there is no hook for that atm. But I am open to add any hooks that have a valid use case.

@keggatron_gitlab like with any nosql database, you have no transactions to enforce this. You have to design your data in away that work with that. For example store the relation data directly in the user object.

Keegan

@keggatron_gitlab

@pubkey so for some entities that's not possible. Is a workaround for this to skip those entities that are child entities on initial sync, then update their local db entries then retrying sync again? Do you know of any examples of projects that have dealt with the same issue?

Martin Foerster

@phal0r

@pubkey I want to build a search plugin based on fuse.js. It also allows to persist the generated index. It needs to be updated on document changes/insertes/deletes of course and also on migrations, but since migration is a batch operation I guess, that it will become inperformant to update the index on each doc, but instead ditch the index, when migration starts and recreate it once migration is finished.

Daniel Meyer

@pubkey

@phal0r can you make a PR with the hooks that you need?

Daniel Meyer

@pubkey

@keggatron_gitlab no sorry I have no examples. yes you can do non-live replications to have some kind of 'checkpoints' where you know that everything is replicated

Martin Foerster

@phal0r

@pubkey Yeah, should not be a problem

ecrofpu

@ecrofpu

Hey folks, I am again here to trouble you by asking a question.

I am facing a situation where I have observed that while pushing a document if there is any error in the response of API then the new documents that get added later will not be pushed as the document with erroneous response in API is not synced.

So I was wondering how to deal with such a situation programmatically? I thought deleting such a document with remove() method should let newer documents get pushed but I have found that RxDB still tries to push the same document. So what can I do here?

I am using the GraphQL replication plugin.

Daniel Meyer

@pubkey

@ecrofpu RxDB will retry the document until the server gives no error. If the server does not want to save the document, it should still return a valid response

Anton Melser

@AntonOfTheWoods

Ok, maybe I'm completely n00b still, but I just had an amazingly bizarre problem:

Uncaught Error: missing node with id ¡
    at minimalStringToSimpleBdd (options-bundle.js:2829)
    at Module../node_modules/event-reduce-js/dist/es/bdd/bdd.generated.js (options-bundle.js:7712)
    at __webpack_require__ (options-bundle.js:42288)
    at Module../node_modules/event-reduce-js/dist/es/index.js (options-bundle.js:7752)
    at __webpack_require__ (options-bundle.js:42288)
    at Module../node_modules/rxdb/dist/es/event-reduce.js (options-bundle.js:25943)
    at __webpack_require__ (options-bundle.js:42288)
    at Module../node_modules/rxdb/dist/es/rx-query.js (options-bundle.js:32795)
    at __webpack_require__ (options-bundle.js:42288)
    at Module../node_modules/rxdb/dist/es/plugin.js (options-bundle.js:26211)

This is for an options page in a chrome extension. I originally had very basic HTML<head> but want to be able to run the extensions on Android so I had copy pasted html5 headers but also left what I had before, resulting in a borked head that worked. I did a bit of a cleanup and started getting this error. After two hours later, going through every single line that was changed between the two, I finally said "well if it is identical and works then I don't care" so made the final change to clean up the HTML <head>, and discovered that the reason for this error was removing a <meta charset="utf-8"> in the HTML of my options page. WTF?!? How can this be even remotely related to anything at all????

ecrofpu

@ecrofpu

@pubkey thanks for the reply.

Daniel Meyer

@pubkey

@/all

Released version 9.16.0

Features:

Added RxDatabase.migrationStates() which returns an observable to observe the state of all ongoing migrations.
Added startupPromise to the returned object of RxDatabase().server()

https://twitter.com/rxdbjs/status/1381724727018991620

Sage

@neonfuz

I wanted to try using svelte and rxjs together but ran into some issues with rollup. So I forked sapper-webpack and added rxdb, here's my result, feedback appreciated:

https://github.com/neonfuz/rxsapper

And on that note, I tried to get rxjs working with both sveltekit and sapper-rollup and have not had success. At this point I've given up on rollup and switched to webpack.

Jurijs Kovzels

@jurijsk

Hi there again. I was posting this a while back but did not get any replies. I will try again if you do not mind. I'm considering using RxDB with GraphQL. I expect that the user will have multiple tabs open (browser extension) plus collaborative editing with other users.
I've read https://rxdb.info/replication-graphql.html page but I'm not sure that I grasp everything it says. Firstly, does replication means that the whole database will be replicated on the client-side? Secondly, in 'cons' it says: "It is assumed that the GraphQL-server is the single source of truth" - for me it sounds like a good thing, why is it a minus? Thank you for your time! Also, I wonder if there are any tutorials on creating a historical database (not sure about the term) with Rxdb, meaning that nothing is removed each edit leads to the new version, and the user can potentially review revert the changes.

Martin Foerster

@phal0r

Firstly, does replication means that the whole database will be replicated on the client-side?

The graphql endpoint can return any data you want. This can be constrained by time for example. The way the graphql sync is set up, is like consuming a data feed.

Secondly, in 'cons' it says: "It is assumed that the GraphQL-server is the single source of truth" - for me it sounds like a good thing, why is it a minus?

RxDB uses pouchdb under the hood, which is protocol-compatible with couchdb. This technology can sync both ways as it keeps a concatenated chain of changes. It is comparable to how git and commits work. In contrast graphql sync needs to resolve all conflicts server-side and decide there, what should happen with the data.

tomelic

@tomelic

Hi there,
was wondering - how severe is it that I organize nested nested nested nested nested data inside a collection schema.. 😅

For example, I have an array of objects of whom some keys are arrays of objects... and so on.

Will this break the program?

And are there ANY examples of accessing such nested fields? or is it evil to even do? (almost nothing on the docs on esoteric nested data handling)

richakumari

@RichaKumari1

Hi @pubkey ,

I have implemented encryption of all fields in RxDB collection's schema file in my electron app, using following libraries with the version mentioned
1.RxDB (9.5.0)
2.pouchdb-adapter-leveldb (7.2.1)
3.leveldown (5.5.1)

Now when I delete the previously created RxDB's, and create new ones I get this error:

TypeError: Cannot set property '_rev' of undefined
[1] at D:\comproDLS\nemo-micro-foc\node_modules\rxdb\dist\lib\rx-collection.js:313:26
[1] at Array.map (<anonymous>)
[1] at D:\comproDLS\nemo-micro-foc\node_modules\rxdb\dist\lib\rx-collection.js:311:39
[1] at processTicksAndRejections (internal/process/task_queues.js:86:5)
[1] at async D:\comproDLS\nemo-micro-foc\desktop\local-database\operations\ebooks-operator.js:60:26
[1] at async EventEmitter.listener (D:\comproDLS\nemo-micro-foc\node_modules\electron-better-ipc\source\main.js:60:22)
[1] [26328:0414/131621.274:INFO:CONSOLE(122)] "Get Ebooks", source: D:\comproDLS\nemo-micro-foc\desktop\web-preload.js (122)
[1] [26328:0414/131621.625:INFO:CONSOLE(1545)] "ERROR", source: webpack-internal:///./node_modules/@angular/core/esm5/core.js (1545)

and on terminating and then on creating again the db's I run into this error:

{ RxError (COL19): RxError:
[1] Pouchdb document update conflict
[1] Given parameters: {
[1] id:"U2FsdGVkX1+8UKDUqT9X9P34khxuXWeMafzsYXs8wA54+3fw3t09xhZ3MJs7lOCXKB0ByvG9Zyc+SwlkABD2kw=="
[1] pouchDbError:{
[1] "status": 409,
[1] "name": "conflict",
[1] "message": "Document update conflict",
[1] "error": true,
[1] "id": "U2FsdGVkX1+8UKDUqT9X9P34khxuXWeMafzsYXs8wA54+3fw3t09xhZ3MJs7lOCXKB0ByvG9Zyc+SwlkABD2kw==",
[1] "docId": "U2FsdGVkX1+8UKDUqT9X9P34khxuXWeMafzsYXs8wA54+3fw3t09xhZ3MJs7lOCXKB0ByvG9Zyc+SwlkABD2kw=="
[1] }

Note: I tried fixing this using this while returing collection's schema : JSON.parse(JSON.stringify(data)).
But it didn't worked.

Daniel Meyer

@pubkey

@tomelic yes this is no problem. I also do that. Then only unsolved thing is how to fix the typings to access generated stuff in nested objects. like myDoc.foo.bar.users$
@RichaKumari1 this looks like a bug, please make a PR with a failing test case. But I think we need to upgrade the electron examples first

@/all

Released version 9.17.0

You can now change, update, remove and add RxAttachments inside of a migration strategy.
https://twitter.com/rxdbjs/status/1381724727018991620

Andrey Stepankov

@rougsig

Hey folks. I need to useanyOf logic. If im sure, its not implemented yet pubkey/rxdb#182

I need to store a set of different object types in one array. How way i can do it, with rxdb? Maybe some workarounds can be there?

Daniel Meyer

@pubkey

anyOf is not supported atm

Daniel Meyer

@pubkey

@rougsig when you set type: "object" to a property, it is like the any type in typescript. there you can then store whatever you want

Where communities thrive

People

Repo info

Activity

Released version 9.16.0

Released version 9.17.0