These are chat archives for y-js/yjs

10th
Sep 2017
Joe Attard-Owen
@Joeao
Sep 10 2017 04:19 UTC
Can I use IndexedDB to populate the contents of my Quill instances with locally stored data before YJS is synced up? Was thinking of storing contents in localstorage but maybe it's not necessary
Winston Fassett
@WinstonFassett
Sep 10 2017 14:06 UTC
@NathanaelA yes I'm doing something similar with setTimeout and "dirty" room tracking. For closing rooms, I keep track of the connection count per room and close the room when it goes to zero, with a short delay in case someone reconnects.
When I process dirty rooms, I create a new YJS connection passing it a config of the shares I want to extract (because the currently open YJS instance for the room does not get created with a share config). I use that connection with a helper function that exports collab object content to JSON.
It's worked pretty well, but I have had lots of problems with corruption, particularly when I edit with my phone and desktop at the same time.
However, I'm starting to think that my corruption problems are not related to the way I'm extracting content with additional YJS rooms
Winston Fassett
@WinstonFassett
Sep 10 2017 14:11 UTC
I disabled all the content exporting and dirty checking and still had the same problems
I switched back to using "memory" as the database (rather than leveldb) and things got a lot better
but my rooms don't persist
Winston Fassett
@WinstonFassett
Sep 10 2017 14:18 UTC
it's occurred to me that I could use the same hooks I already have to save the ops off to a database, which raises similar questions to those of @NathanaelA around how I could initialize a room from the ops in the database before serving it to clients
I also noticed @dmonad working on persistence features in yjs and redis in y-websockets-server, so I'm wondering how leveldb fits into that future, if at all
Winston Fassett
@WinstonFassett
Sep 10 2017 15:27 UTC
y-js/yjs@8770c8e
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:25 UTC
@WinstonFassett - That is an interesting commit. At the moment my last test last night I came up with a slightly different design. Initially I was going to use a room per file; but I realized that my apps breakdown better matches a room per workspace; so instead of persisting a single edit inside the room, I have created a files "Map". Then I create new YTexts that I attach and remove from the files Map. The interesting thing is that by switching to this method my persistence design started working properly. I have tested it several times with several browsers, editing; reloading the browser, resetting the server during the session and everything seems to be great. I kinda feel like I'm missing the other shoe... ;-)
I'm create a YJs using the Memory db on both client and server, and using websockets for the communication.
When the server gets a request for a room that doesn't exist, it creates it and pre-fills in the data like this.
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:30 UTC
function getInstanceOfRoom(room) {
// if workspace doesn't exist ...
workspace_files_rooms[room] = new Promise((resolve) => {
                Y({
                    db: {
                        name: "memory",
                        namespace: room
                    },
                    connector: {
                        name: 'websockets-server',
                        room: room,
                        io: io,
                        debug: !!options.debug
                    },
                    share: {
                        files: 'Map',
                    }
                }).then(function (y) {
                   // Example of creating a file, "default" is the file key
                    y.share.files.set("default", Y.Text);
                    y.share.files.get("default").insert(0, "Creating room" + room);
                    resolve(y);
                });
  return workspace_files_room[room];
On the client it is also a memory db, and so what happens is the y.share.files.set(<filename>, Y.Text) which the server handles and then the following line y.share.files.get(<fileName>).insert(0, prefill_data) will prefill the data on creation of the room.
For persistence of the data; I do this:
```
 socket.on('yjsEvent', function (msg) {
            if (msg.room != null) {
                getInstanceOfRoom(msg.room).then(function (y) {
                    y.connector.receiveMessage(socket.id, msg);
                    if (msg.type === 'update') {
                        workspace_tracking[msg.room].saveTimeout = Date.now()+3000;
                    }
                });
            }
        });
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:36 UTC
I have a timer that runs every second and goes through each room and sees if a room needs to be saved or deleted. If saved, then I grab the current value of the files and persist it in my persistence layer.
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:44 UTC
To my knowledge this works. I've been playing with it since late last night and I am able to initialize the rooms on demand with fake data. Going to integrate it into my project and see if it actually works the way my demo does; but it does seem to be able to persist and restore a room correctly, and in the event my yJS server worker crashes, it is able to come back up and the data on the browser seems to be treated as newer (at the moment they type).
Issue: Server crashes, and comes back up. Server is considered source of truth and could have lost up to 4 seconds of data. If the person in the browser hasn't typed anything since the server came back up; those changes will be lost. I'm not seeing a easy way to recover the browser data until they type something. I think this is acceptable; the server shouldn't crash and odds are very likely if they were typing when the server crashed and came back up they probably are still typings. But there is a small window here where I could lose a small amount of data.
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:50 UTC
@dmonad - If you have any ideas or see any issues with this design, it seems to work great other than that small 3-4 second potential for dataloss, which I see as acceptable risk because it should be a very very rare case.
Nathanael Anderson
@NathanaelA
Sep 10 2017 18:56 UTC
I should probably clarify, if the server comes up and they are still typing, they lose no data. The server side automatically is updated to the browsers full version of the changes. So, the corner case is a pretty specific "Type, crash, type, server comes up, no more typing = data loss of the data typed during the crash."