Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Danny Andrews
    @danny-andrews
    Btw, upgrading to yjs@13.4.6 and y-websocket@1.3.8 fixed all my memory leak issues AFAICT, and no more reconnects every 30 seconds after restarting my server!
    Kevin Jahns
    @dmonad
    @danny-andrews It's all in the docs: https://docs.yjs.dev/getting-started/adding-awareness Also the API documentation: https://docs.yjs.dev/api/about-awareness Still incomplete, but I'm working on it...
    Danny Andrews
    @danny-andrews
    Whoops. I've been looking at the README. Thanks!
    hfroger
    @froger

    Hello, I am exploring yjs and started to use a bit with our slate editor, work out-of-box that's great! I am amazed how it works fast, I am wondering now if I can use yjs to actually make apps (like room.sh).

    I am wondering if there are any architecture advices to build apps CRUD based, or open-source examples. To me, it is unclear:

    • Best practice to handle relationship between models (subdocuments?)
    • Best practice to handle sorting (re-ordering array indexes, building a separate array just for listing references ?)
    • Best practice to handle permissions on document, if it is possible?

    Thanks for any pointer !

    1 reply
    Kevin Jahns
    @dmonad

    You can certainly use Yjs to build apps. Room.sh, and many other apps, use Yjs as a data model.

    The providers handle authentication. y-websocket allows you to handle auth over cookies or request parameters. y-webrtc encrypts documents using a shared password. Yjs itself is encryption/auth-agnostic. You can implement your own auth mechanism.

    The documentation https://docs.yjs.dev is improving rapidly. If you want me to talk about how companies use Yjs, you can hit me up with a sponsorship and you can pick my brains in private sessions. There is also https://discuss.yjs.dev/ where you can ask specific questions.

    1 reply
    erdzan12
    @erdzan12
    When trying to import YJS with import * as Y from 'yjs' inside of a Polymer LitElement project, an error occurs in the browser saying Uncaught SyntaxError: The requested module '../isomorphic.js/iso.js' does not provide an export named 'default'. Is polymer not supported by default? I tried the solution of @istvank by starting the es-dev-server with --node-resolve but this does not seem to help.
    Gabriel Rogan
    @gaberogan

    Hey @dmonad, I'm trying to get YJS working with AWS API Gateway Websockets with a Node.js Lambda since lamba supports websockets as of 2018 and it seems like the most scalable solution for my CMS use-case. Do you have any advice as to what it would take to get this working?

    I'm thinking I'll basically need to update y-websockets-server by pulling out socket.io and replacing it with the AWS stuff, and then moving local state such as 'rooms' to redis (AWS Redis global datastore) to keep the lambdas stateless. Also if you think this is a bad idea, I would appreciate any advice there as well.

    Gabriel Rogan
    @gaberogan
    Edit: y-websockets-server seems outdated and not what I want. I just realized y-websocket has a server implementation. So I'm thinking I'll write a light aws wrapper for native websockets and pull out local state like docs.
    Kevin Jahns
    @dmonad
    Hi @erdzan12 The issue was discussed here: dmonad/lib0#9 Yjs is an isomorphic module (it works in the browser and in nodejs). We sometimes want to import node-native features using commonjs. es-dev-server doesn't handle commonjs modules correctly so you get this error message.
    Kevin Jahns
    @dmonad

    @gaberogan Yjs works well together with redis. You should absolutely do that. In some intervals you probably want to call a worker process that collects the room-data and persists that to a persistent database.

    The y-websockets repository contains the current server implementation (see y-websocket/bin/server.js). I think it makes sense to use that instead of writing another wrapper. You can use the persistence hook to integrate with redis.

    1 reply
    Tom Enden
    @tomenden

    Hi, I'm looking at the option of using Yjs for a collaborative code-editor product. Just started going through the tutorials, and I'm wondering if it can fit my usecase. Some background info - The product already contains an external DB with a lot of documents (each "document" for the purpose of this message is a full fileSystem representation), and has its own Authorization logic.
    Looking into https://github.com/yjs/y-websocket it seems that the server is meant to be used "as-is" (i.e I see the recommendation is to run PORT=1234 npx y-websocket-server). What would be the right way to customize this in order to allow custom authentication? Should I copy the code from .bin/server.js and modify it or can I somehow require the server part in node? I also saw that the recommendation is to keep the data in Yjs format. Seeing as this is a mature product with A LOT of users & documents, data-migration is not an option, and the system will always need to be backwards compatible. Therefore - I would like to keep the data in its current format (in which each file content is just a simple string). What are the pitfalls to that? Would it be considered a blocker in your opinion?

    High level architecture I had in mind - A room (if I understand the terminology correctly) for each Document (=fileSystem). Allow connection to the room only to authenticated users that have the permission to enter the room. On load (first connection) - initialize the room and the Y.Doc from the data in the DB. Each change will go through the yjs server, and post-update will be persisted to the db (which works in "auto-save" mode, similar to google-doc). WDYT? Any links to relevant resources would be great, as I wasn't able to find resources for what I was looking for.

    Thanks! @dmonad Yjs looks awesome :)

    Kevin Jahns
    @dmonad
    Hi @tomenden ,
    Yjs integrates well into existing communication infrastructures and it supports any kind of database. Some databases are better suited than others (postgres is better suited to store the document in some kind of interval, while NoSQL databases are also suited to store each incremental document update - see y-leveldb) You can use y-websocket as-is or include the components you need (as I do in https://github.com/yjs/yjs-demos/tree/master/demo-server). Depending on your use-case, you might need to fork it.
    There is already a lot of information on how you can build your custom providers here: https://docs.yjs.dev/
    I'm constantly improving the documentation. If you need more help I recommend to sponsor me and select the "support contract" (see https://github.com/sponsors/dmonad). This will get you one session a week with me and will also allow me to improve the documentation further.
    1 reply
    Braden
    @AdventureBeard
    Hey @dmonad , hope you had a happy new year! In your "How Yjs works from the inside" video, you mention that someone made a visualization for the data structure of a Yjs document. Is that available anywhere? Would love to have a look at it!
    Kevin Jahns
    @dmonad
    Hi @AdventureBeard , happy new year!
    https://text-crdt-compare.surge.sh/ - The version that they used had a bug: It doesn't remove formatting markers (e.g. {bold: true}). You can walk through the data structure yourself (for Y.Text: item = y.text._start; item = item.right to iterate)
    1 reply
    Braden
    @AdventureBeard
    Wow, this is fantastic! Thanks!
    Jeremy Dombrowski
    @meatflavourdev
    Is there a demonstration or library that integrates React with Yjs, something like https://github.com/relm-us/svelt-yjs for Svelte-- Might save me some time 🤷‍♂️
    4 replies
    Coder of Salvation / Leon van Kammen
    @coderofsalvation
    hi all, is there a demo for a collaborative dom-elements? I noticed the yjs/y-dom repo, but I can't find any docs or demos about it.
    Kevin Jahns
    @dmonad
    Hi @coderofsalvation I don't maintain the repository anymore. It doesn't work with the current version of Yjs.
    It was a fun experiment, but I recommend to use the editor bindings instead.
    Coder of Salvation / Leon van Kammen
    @coderofsalvation
    ah ok thanks
    Jeremy Dombrowski
    @meatflavourdev
    Can yjs shared types be updated the same way you might setState in React, as if state is immutable-- you modify it and pass it to setState() and it does all the diff'ing and updating for you?
    Kevin Jahns
    @dmonad
    Hi @meatflavourdev
    You are right about your first question. Every object in Yjs can be uniquely identified. So when you delete something, Yjs will make sure that only the intended item is deleted.
    Second question: No you can't update elements in Y.Array (yet - I plan to implement this feature eventually). For now you have to delete & insert.
    Third: The diffing is not part of Yjs, but can be implemented on-top of Yjs. ProseMirror works on immutable state. y-prosemirror calculates efficient diffs on the Yjs model. But you can use something like zustand-yjs if you are working with react (or implement your own store with Yjs).
    Jeremy Dombrowski
    @meatflavourdev
    perfect thanks ;) @dmonad
    So if I were to delete and insert and I wanted it to be like an update-- I should use a transaction?
    Kevin Jahns
    @dmonad

    That would make sense. Just keep in mind that if another user "set" the same value at the same time, you will end up with both inserted values in the list. The semantics of list.set(i, updatedValue) would enforce that only one change would prevail.

    If you really need to update values in a list, you can embed a Y.Map in the Y.Array (Y.Array<Y.Map>). Then you could do something like yarray.get(i).set('value', "my updated value")

    Jeremy Dombrowski
    @meatflavourdev

    You can embed a Y.Map in the Y.Array (Y.Array<Y.Map>). Then you could do something like yarray.get(i).set('value', "my updated value")

    Very helpful, thank you.

    Gabriel Rogan
    @gaberogan
    @dmonad Here's an open source implementation + demo of YJS with DynamoDB + AWS Websocket API I made. I've been getting multiple emails from my earlier message on this thread, so it seems like people are interested in this. I hear you're also working on a stateless y-websocket server, so it'll be interesting to see how you handle that differently.
    https://github.com/gaberogan/y-websocket-api
    Winston Fassett
    @WinstonFassett
    @gaberogan thanks for this! The demo works great. I look forward to trying it.
    I looked at the handler and had some ideas on optimizing this which I might try if I get time.
    I think you can save a lot of DynamoDB data loading and lambda cycles per ws message by avoiding calling getOrCreateDoc unless you really need it.
    For example with messageAwareness messages, you can just broadcast and return early because they don't mutate the doc.
    Even with messageSync messages, you may only really need to load the doc for writeSyncStep1.
    That is to say, you might be able to skip loading the doc even for updates, because you are just doing a push and a broadcast
    I see that you are calling Y.applyUpdate but it doesn't seem like that is necessary when just pushing an update to dynamo and rebroadcasting the original message
    I think with those changes you would only ever have to actually read from DynamoDB and load the document and apply all known updates when clients send writeSyncStep1 after connecting
    Kevin Jahns
    @dmonad
    I can imagine that a lot of people are interested in something like this! Thanks for sharing :) The y-websocket backend is not satisfactory out-of-the-box in most cases, so it is great that you are working on something that can be easily installed on AWS.
    @WinstonFassett has some valuable feedback. Loading Yjs documents is the most expensive operation in y-websocket server. So my approach is to avoid loading the document by computing the differences directly on the binary update messages using the API I proposed here: yjs/yjs#263
    The PR for this change is ready, I will review it this weekend and publish. You should have a look at it if you want to improve the performance of your implementation. It shouldn't be too much effort to adapt to the new approach.
    It would be great if you could publish your work on https://discuss.yjs.dev/ if you want to make it available (using the "show" tag). It doesn't get too many visitors, but it is nice to keep track of the work that people are doing and have been doing in the past.
    Eduard Bardají Puig
    @aryzing
    Any suggestions on how to use yjs to keep a distributed graph in sync?
    Kevin Jahns
    @dmonad
    You can model a graph using Yjs' shared types. One example how you can model a graph is https://discuss.yjs.dev/t/example-of-yjs-used-to-share-an-editable-network-graph-with-vis-network/105/2 from @micrology
    Eduard Bardají Puig
    @aryzing
    That's a great resource, thanks
    Nigel Gilbert
    @micrology
    @aryzing And you can find the app that uses this code at https://cress.soc.surrey.ac.uk/prsm/ (that's the landing page which leads you to the app itself and the GitHub repository - it's open source). It's still in active development and comments are very welcome.
    Nigel Gilbert
    @micrology
    I'm looking for sample code that, given a Y.doc that includes a Y.Map, allows me to extract the history of all insertions and deletions (i.e something like [{clock, key, value}, {clock, key, value}, ...]), starting from when the Y.doc was created to the current state, where 'key' and 'value' are what was provided to the set method of the Y.Map. Any pointers?
    Kevin Jahns
    @dmonad
    That's not really possible. You can, however, capture the history of updates and apply them one-by-one to recreate the editing history.
    Thomas Osmonson
    @aulneau
    Sorry if this has been asked before -- i tried to search but couldn't find exactly this situation. I am building an editor based in prosemirror and I'm syncing the data to the remote server. I'm curious what the most effective way to determine if there are diffs between the two (client persisted with indexeddb, and server)? the remote data is a hex string of the resulting buffer from Y.encodeStateAsUpdate. I've noticed that when the editor syncs, there is always an initial change. for my yDoc.on('update') handler, i'm trying to avoid a write to the server that is from the initial sync plugin loading event, eg when we already have the state, and the sync has just set the content (meaning they will be equal)
    Kevin Jahns
    @dmonad
    Hi @aulneau , could you elaborate what you are trying to achieve? What kind of diff do you want to compute?
    Andrew Milich
    @amilich
    "That's not really possible. You can, however, capture the history of updates and apply them one-by-one to recreate the editing history." @dmonad any reference to docs or an example here?
    Thomas Osmonson
    @aulneau

    Hi @aulneau , could you elaborate what you are trying to achieve? What kind of diff do you want to compute?

    Sure! Basically I have local state for my docs, persisted with the indexeddb provider, but I also persist the docs on an interval to remote storage. When a user starts the app and loads a document, local state is used first to load the document, and when the app fetches the remote version from storage, it will apply it to the doc as an update. I want to check the remote version of the doc against the locally persisted doc, and if there are no diffs between them, not apply the update. Is that more helpful?

    Kevin Jahns
    @dmonad
    @amilich The updates API is documented: https://docs.yjs.dev/api/document-updates You could listen to the 'update' event to listen to incremental updates and then call Y.applyUpdate(freshYDoc, update) on the updates that were created before a certain point in time.
    @aulneau I assume that the remote version is also in the Yjs update format (not some kind of JSON-prosemirror document). In this case you can simply always apply the document update. The local document won't fire change-events if there are no differences.
    Thomas Osmonson
    @aulneau
    Thank you :) @dmonad I have one more question if that's ok. I have a yDoc that contains some maps, one for meta data, one for the actual content of the item. Some of the entries in the map are strings. Lets look at the title entry -- it's just a string, and is updated as such. Would it be better for the string elements to be of type Y.Text? and if so, how would I bind an input/textarea to the instance?
    I've found this -- https://github.com/y-js/y-text but it's very old and the latest Y.Text doesn't have any of these bind methods
    Kevin Jahns
    @dmonad

    Hi @aulneau If you want the title to support concurrent text-insertions, then you should use the Y.Text type. But IMO the title field doesn't necessarily need to support that.

    A binding basically works like this: When the Yjs model changes (in this case the Y.Text or the Y.Map title field) then you update the text entry and update the selection to restore the previous selection. When the input area changes, then you update the model. You should prevent endless loops. I like to use lib0/mutex or the origin of the transaction. A simple boolean flag will suffice.

    The old y-text type did natively support input-fields. The implementation is fairly complex as you need to listen to the dom API. I didn't port the old API yet (I already created a repository for it http://github.com/yjs/y-textarea).

    So you could now start with implementing your own input-binding that listens to granular changes (e.g. string 'xxx' was inserted at pos x) or simply replace the whole content every time anything changes (using Y.Map).

    Thomas Osmonson
    @aulneau
    @dmonad perfect, thanks! that makes a lot of sense. I'll stick with just overwriting it :) There is one other thing I am currently trying to tackle and i can't come up with any solutions via searching/digging around. I am using a prosemirror based editor (rich-markdown-editor) and I have it hooked up and working well with yjs. However, the editor supports inserting different type of nodes, one being an embed. I have found when I have the editor hooked up to yjs and I insert one of these embeds, I get an error from AbstractType.js of AbstractType.js:826 Uncaught Error: Unexpected content type. I have been digging and digging and I just don't think I know enough about how it's implemented behind the scenes. Does this ring any bells for you? If there's anything you can point me to to play around with, I'd really appreciate it :)
    Kevin Jahns
    @dmonad
    You might have two different versions of Yjs installed.
    If that is not the case, you could inspect the type of content that is inserted into the Yjs document and post that here.
    ilovedesert001
    @ilovedesert001
    Hello, anyone?
    a strange problem.
    After storing 10,000 + node data, then reloading , remain data less than 10000 ...
    why ? is this expected ?
    add 10000 data to map
    ilovedesert001
    @ilovedesert001
    Solved, use transaction