Hello, I am exploring yjs and started to use a bit with our slate editor, work out-of-box that's great! I am amazed how it works fast, I am wondering now if I can use yjs to actually make apps (like room.sh).
I am wondering if there are any architecture advices to build apps CRUD based, or open-source examples. To me, it is unclear:
- Best practice to handle relationship between models (subdocuments?)
- Best practice to handle sorting (re-ordering array indexes, building a separate array just for listing references ?)
- Best practice to handle permissions on document, if it is possible?
Thanks for any pointer !
1 reply
You can certainly use Yjs to build apps. Room.sh, and many other apps, use Yjs as a data model.
The providers handle authentication. y-websocket allows you to handle auth over cookies or request parameters. y-webrtc encrypts documents using a shared password. Yjs itself is encryption/auth-agnostic. You can implement your own auth mechanism.
The documentation https://docs.yjs.dev is improving rapidly. If you want me to talk about how companies use Yjs, you can hit me up with a sponsorship and you can pick my brains in private sessions. There is also https://discuss.yjs.dev/ where you can ask specific questions.
1 reply
import * as Y from 'yjs' inside of a Polymer LitElement project, an error occurs in the browser saying Uncaught SyntaxError: The requested module '../isomorphic.js/iso.js' does not provide an export named 'default'. Is polymer not supported by default? I tried the solution of @istvank by starting the es-dev-server with --node-resolve but this does not seem to help.
Hey @dmonad, I'm trying to get YJS working with AWS API Gateway Websockets with a Node.js Lambda since lamba supports websockets as of 2018 and it seems like the most scalable solution for my CMS use-case. Do you have any advice as to what it would take to get this working?
I'm thinking I'll basically need to update y-websockets-server by pulling out socket.io and replacing it with the AWS stuff, and then moving local state such as 'rooms' to redis (AWS Redis global datastore) to keep the lambdas stateless. Also if you think this is a bad idea, I would appreciate any advice there as well.
@gaberogan Yjs works well together with redis. You should absolutely do that. In some intervals you probably want to call a worker process that collects the room-data and persists that to a persistent database.
The y-websockets repository contains the current server implementation (see y-websocket/bin/server.js). I think it makes sense to use that instead of writing another wrapper. You can use the persistence hook to integrate with redis.
1 reply
Hi, I'm looking at the option of using Yjs for a collaborative code-editor product. Just started going through the tutorials, and I'm wondering if it can fit my usecase. Some background info - The product already contains an external DB with a lot of documents (each "document" for the purpose of this message is a full fileSystem representation), and has its own Authorization logic.
Looking into https://github.com/yjs/y-websocket it seems that the server is meant to be used "as-is" (i.e I see the recommendation is to run PORT=1234 npx y-websocket-server). What would be the right way to customize this in order to allow custom authentication? Should I copy the code from .bin/server.js and modify it or can I somehow require the server part in node? I also saw that the recommendation is to keep the data in Yjs format. Seeing as this is a mature product with A LOT of users & documents, data-migration is not an option, and the system will always need to be backwards compatible. Therefore - I would like to keep the data in its current format (in which each file content is just a simple string). What are the pitfalls to that? Would it be considered a blocker in your opinion?
High level architecture I had in mind - A room (if I understand the terminology correctly) for each Document (=fileSystem). Allow connection to the room only to authenticated users that have the permission to enter the room. On load (first connection) - initialize the room and the Y.Doc from the data in the DB. Each change will go through the yjs server, and post-update will be persisted to the db (which works in "auto-save" mode, similar to google-doc). WDYT? Any links to relevant resources would be great, as I wasn't able to find resources for what I was looking for.
Thanks! @dmonad Yjs looks awesome :)
Yjs integrates well into existing communication infrastructures and it supports any kind of database. Some databases are better suited than others (postgres is better suited to store the document in some kind of interval, while NoSQL databases are also suited to store each incremental document update - see y-leveldb) You can use y-websocket as-is or include the components you need (as I do in https://github.com/yjs/yjs-demos/tree/master/demo-server). Depending on your use-case, you might need to fork it.
There is already a lot of information on how you can build your custom providers here: https://docs.yjs.dev/
I'm constantly improving the documentation. If you need more help I recommend to sponsor me and select the "support contract" (see https://github.com/sponsors/dmonad). This will get you one session a week with me and will also allow me to improve the documentation further.
1 reply
https://text-crdt-compare.surge.sh/ - The version that they used had a bug: It doesn't remove formatting markers (e.g.
{bold: true}). You can walk through the data structure yourself (for Y.Text: item = y.text._start; item = item.right to iterate)
1 reply
4 replies
You are right about your first question. Every object in Yjs can be uniquely identified. So when you delete something, Yjs will make sure that only the intended item is deleted.
Second question: No you can't update elements in Y.Array (yet - I plan to implement this feature eventually). For now you have to delete & insert.
Third: The diffing is not part of Yjs, but can be implemented on-top of Yjs. ProseMirror works on immutable state. y-prosemirror calculates efficient diffs on the Yjs model. But you can use something like
zustand-yjs if you are working with react (or implement your own store with Yjs).
That would make sense. Just keep in mind that if another user "set" the same value at the same time, you will end up with both inserted values in the list. The semantics of list.set(i, updatedValue) would enforce that only one change would prevail.
If you really need to update values in a list, you can embed a Y.Map in the Y.Array (Y.Array<Y.Map>). Then you could do something like yarray.get(i).set('value', "my updated value")
https://github.com/gaberogan/y-websocket-api
I looked at the handler and had some ideas on optimizing this which I might try if I get time.
I think you can save a lot of DynamoDB data loading and lambda cycles per ws message by avoiding calling
getOrCreateDoc unless you really need it.For example with
messageAwareness messages, you can just broadcast and return early because they don't mutate the doc.Even with
messageSync messages, you may only really need to load the doc for writeSyncStep1.That is to say, you might be able to skip loading the doc even for updates, because you are just doing a push and a broadcast
I see that you are calling
Y.applyUpdate but it doesn't seem like that is necessary when just pushing an update to dynamo and rebroadcasting the original messageI think with those changes you would only ever have to actually read from DynamoDB and load the document and apply all known updates when clients send writeSyncStep1 after connecting
@WinstonFassett has some valuable feedback. Loading Yjs documents is the most expensive operation in y-websocket server. So my approach is to avoid loading the document by computing the differences directly on the binary update messages using the API I proposed here: yjs/yjs#263
The PR for this change is ready, I will review it this weekend and publish. You should have a look at it if you want to improve the performance of your implementation. It shouldn't be too much effort to adapt to the new approach.
It would be great if you could publish your work on https://discuss.yjs.dev/ if you want to make it available (using the "show" tag). It doesn't get too many visitors, but it is nice to keep track of the work that people are doing and have been doing in the past.
Y.encodeStateAsUpdate. I've noticed that when the editor syncs, there is always an initial change. for my yDoc.on('update') handler, i'm trying to avoid a write to the server that is from the initial sync plugin loading event, eg when we already have the state, and the sync has just set the content (meaning they will be equal)
Hi @aulneau , could you elaborate what you are trying to achieve? What kind of diff do you want to compute?
Sure! Basically I have local state for my docs, persisted with the indexeddb provider, but I also persist the docs on an interval to remote storage. When a user starts the app and loads a document, local state is used first to load the document, and when the app fetches the remote version from storage, it will apply it to the doc as an update. I want to check the remote version of the doc against the locally persisted doc, and if there are no diffs between them, not apply the update. Is that more helpful?
Y.applyUpdate(freshYDoc, update) on the updates that were created before a certain point in time.
title entry -- it's just a string, and is updated as such. Would it be better for the string elements to be of type Y.Text? and if so, how would I bind an input/textarea to the instance?
Y.Text doesn't have any of these bind methods
Hi @aulneau If you want the title to support concurrent text-insertions, then you should use the Y.Text type. But IMO the title field doesn't necessarily need to support that.
A binding basically works like this: When the Yjs model changes (in this case the Y.Text or the Y.Map title field) then you update the text entry and update the selection to restore the previous selection. When the input area changes, then you update the model. You should prevent endless loops. I like to use lib0/mutex or the origin of the transaction. A simple boolean flag will suffice.
The old y-text type did natively support input-fields. The implementation is fairly complex as you need to listen to the dom API. I didn't port the old API yet (I already created a repository for it http://github.com/yjs/y-textarea).
So you could now start with implementing your own input-binding that listens to granular changes (e.g. string 'xxx' was inserted at pos x) or simply replace the whole content every time anything changes (using Y.Map).
AbstractType.js of AbstractType.js:826 Uncaught Error: Unexpected content type. I have been digging and digging and I just don't think I know enough about how it's implemented behind the scenes. Does this ring any bells for you? If there's anything you can point me to to play around with, I'd really appreciate it :)
