Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Kevin Jahns
    @dmonad

    Hi @AdventureBeard , transforming a YXmlFragment should be pretty fast. From my experience, what takes a while is to load the ProseMirror document as it involves working with the dom.

    Could you please share some performance profiling information? If only createNodeFromYElement takes 2.5 seconds, then there seems to be a problem.

    Another problem of the Atlaskit react component is that - for whatever reason - the document is initialized several times. schema validation, rendering, decoration handling is all executed a few times when the editor is loaded. I have been struggling with Atlaskit in another project as well. Maybe you could check if you are experiencing something similar.

    Offloading the ydoc⇒PM calculation is probably not an option since this would require an asynchronous calculation and you would need to load the ydoc in the worker thread as well (adding more overhead as needed).

    Nigel Gilbert
    @micrology
    @dmonad There is a typo in the Awareness and Presence section of the Getting Started document. In the code snippet in the Quick Start section, there is a reference to awareness.setLocaleAwarenessField which should be awareness.setLocalStateField.
    2 replies
    Florian Wiech
    @florianwiech
    Hi, @dmonad wrote in his reply (https://discuss.yjs.dev/t/attribution-of-changes/291/4?u=flow) to use the item id in prosemirror instead of generating an own one.
    Has anyone an idea how I can access this item id in prosemirror?
    Oh, and another question in the area of prosemirror and yjs:
    Is is possible to get something like a sync event, when yjs successfully inserted content the first time in the prosemirror state, so that I can start a scroll animation to the desired position?
    (At the moment I start the scroll animation after a timeout with does not feel right. I could not find any sync event in the code...)
    Kevin Jahns
    @dmonad
    Thanks for the hint @micrology I already fixed it. I also added a "progress ticket" where you can give feedback. I'll try to update my progress and my plans in this ticket.
    @florianwiech I'm thinking more about this solution.. I think the best approach is to simply generate a new one. Accessing the item.id from Prosemirror is really hard and overly complicated. You basically would need to compute the ProseMirror path and follow that path in the Yjs document structure. It is probably really not worth it.
    Kevin Jahns
    @dmonad

    For the event "inserting content for the first time" you could simply observe the Yjs type and wait for it to be manipulated.

    const listener = () => {
      console.log('inserted content for the first time')
      yxmlFragment.unobserve(listener)
    }
    yxmlFragment.observe(listener)

    There is also a synced event that is called on the provider when it first syncs with another client.

    2 replies
    The listener gets an event object that tells you whether teh event is initiated locally or remotely: event.transaction.local
    Nigel Gilbert
    @micrology

    My client shows avatars for all online peer clients, using y-protocol/awareness and y-websocket. Sometimes, the awareness message drops one of the other clients, followed immediately by adding it again. This seems to happen more often when there are many (>5) clients online, but sometimes occurs even with only two. Here's a sample trace output (logged from awareness.on('change', ...)and awareness getStates():

    21:10:42.486 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
    prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
    prsm.js:2716 21:10:42.488 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}
    prsm.js:271 {added: Array(0), updated: Array(0), removed: Array(1)}
    prsm.js:2716 21:11:14.487 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
    prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
    prsm.js:2716 21:11:14.508 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}
    prsm.js:271 {added: Array(0), updated: Array(0), removed: Array(1)}
    prsm.js:2716 21:12:15.495 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
    prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
    prsm.js:2716 21:12:15.496 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}

    (e.g. at 21:11:14.487, client 182058024 was removed. 21 mSecs later, the set of clients was updated to include it again).
    (NB nothing else is happening - none of the clients are broadcasting any activity at all). It is a nuisance because as a consequence the display of avatar icons flashes on each change.
    Is anyone else seeing this sort of behaviour?

    Kevin Jahns
    @dmonad
    It seems that the state timed out and the client is marked as offline. A few seconds later it is added again. What is odd is that the state should time out after 30 seconds, not 23 seconds. Did you tinker with outdatedTimeout (https://github.com/yjs/y-protocols/blob/master/awareness.js#L13) or do you have long propagation times?
    2 replies
    Danny Andrews
    @danny-andrews
    Hey y'all! Has anyone figured out a way to show yDoc syncing status in their UI.
    Actually, nevermind. I just added new IndexeddbPersistence("my-app", doc); and now updates are basically instant...and my app works offline. Incredible.
    I am curious if anyone has run into memory leaks using y-websocket server-side. It's probably something I'm doing wrong but was curious if others had run into issues.
    Nigel Gilbert
    @micrology
    @danny-andrews y-websocket memory leaks: yes - see yjs/y-websocket#24 The last comment in that thread suggests a resolution. I will submit a PR shortly.
    Nandit Mehra
    @nandit123
    Hi @dmonad , i wanted to use an external database to store versions data (from example - prosemirror-versions) and then retrieve it from there when no other peer can serve the document data. Can the versions array containing all the snapshots be used to regenerate the whole document?
    Kevin Jahns
    @dmonad
    The "version-snapshots" are merely pointers to the Yjs document. You use a "snapshot" to reconstruct an old state. You can use two "versions" to compute the differences. This is how the y-prosemirror bindings renders the differences. You can store the snapshots anywhere, but you do need the Yjs document as well.
    Nandit Mehra
    @nandit123
    in the last line, you say that we need the Yjs document as well - but that will be the generic document object structure which can be fed with the snapshot data, right? @dmonad
    also when i do JSON.stringify(versions) to store versions data in a json way, it seems to loose the important version data (like which edits were made by which client). The data i get on doing JSON.stringify(versions) just seems to contain only snapshot time, key-value pair numbered data (maybe this is encoded form of data) and the client id which took the snapshot. Any ideas how to save complete versions data which can be retrieved later to regenerate the document ? @dmonad
    Nandit Mehra
    @nandit123
    Also is there a way to detect if there is no peer to serve a document?
    Kevin Jahns
    @dmonad
    By Yjs document I mean the encoded format of the original document. In the example, I disable "gc" (garbage collection) so that the Yjs document retains the editing history. We use shapshots to restore an old document state (snapshots are just pointers, they don't contain any content).
    To be explicit: You can't "feed" a Yjs document with snapshot data. You need the original content.
    Snapshots (similarly to state vectors) are Uint8Arrays - binary encoded blobs of data. These blobs are not json encodable. You can send them over the via as a binary blob (most transport mechanisms support binary data), or alternatively using base64 encoding. Do not send them using JSON encoding - that would be a waste of bandwith ;)
    Nandit Mehra
    @nandit123
    ok.. thank you @dmonad
    Kevin Jahns
    @dmonad
    NP @nandit123 The whole Snapshot API is undocumented because it is not stable yet. These are all very new and confusing concepts. The snapshot API, relative positions, and state vectors are all concepts that never existed before. So only a few people (familiar with the theory behind it) can understand it. That's not ideal. I'm not going a good job explaining these new concepts to make them accessible. Sorry that they give you a hard time. But I hope all of this will become clearer in the coming months.
    Nandit Mehra
    @nandit123
    yeah i can understand that. Let me know if i can help here.. i have been learning about CRDTs from sometime now and yjs seems useful library.
    Nandit Mehra
    @nandit123
    Hey @dmonad , if i store the current document content and all the snapshots data (which are pointers) in an external storage, do you think if it is possible to regenerate the document as it was?
    I am unable to figure out how can i set the variables of a new Ydoc object if i have the latest doc content and all the previous snapshot data. The aim is to allow new peers to get the data from external storage when there is no other peer online to serve that data to them
    Nandit Mehra
    @nandit123
    and generate the same Yjs doc for them
    Kevin Jahns
    @dmonad

    You can render a specific version by changing the ySyncPlugin state. If you set version to a specific version, you render that version. If you also specify permanentUserData and prevVersion, you can render the differences. https://github.com/yjs/yjs-demos/blob/c3b1fb2a770414ff55b01cb32ac93a0282190ae6/prosemirror-versions/prosemirror-versions.js#L58

    The confusing part is probably that versions are specified on the editor model and not the Yjs document.

    That is because the editor`binding is responsible for calculating the differences. This is why it makes sense to calculate them in the editor binding.
    If you want to reset the current state of the document to an old state, then you simply overwrite the current state of the document with the old versioned state.
    Danny Andrews
    @danny-andrews

    @danny-andrews y-websocket memory leaks: yes - see yjs/y-websocket#24 The last comment in that thread suggests a resolution. I will submit a PR shortly.

    Thanks @micrology

    Danny Andrews
    @danny-andrews
    I see the term "awareness" come up quite a bit in discussions about y-websocket, but can't find any information on what that is. Can anyone fill me in?
    Danny Andrews
    @danny-andrews
    Btw, upgrading to yjs@13.4.6 and y-websocket@1.3.8 fixed all my memory leak issues AFAICT, and no more reconnects every 30 seconds after restarting my server!
    Kevin Jahns
    @dmonad
    @danny-andrews It's all in the docs: https://docs.yjs.dev/getting-started/adding-awareness Also the API documentation: https://docs.yjs.dev/api/about-awareness Still incomplete, but I'm working on it...
    Danny Andrews
    @danny-andrews
    Whoops. I've been looking at the README. Thanks!
    hfroger
    @froger

    Hello, I am exploring yjs and started to use a bit with our slate editor, work out-of-box that's great! I am amazed how it works fast, I am wondering now if I can use yjs to actually make apps (like room.sh).

    I am wondering if there are any architecture advices to build apps CRUD based, or open-source examples. To me, it is unclear:

    • Best practice to handle relationship between models (subdocuments?)
    • Best practice to handle sorting (re-ordering array indexes, building a separate array just for listing references ?)
    • Best practice to handle permissions on document, if it is possible?

    Thanks for any pointer !

    1 reply
    Kevin Jahns
    @dmonad

    You can certainly use Yjs to build apps. Room.sh, and many other apps, use Yjs as a data model.

    The providers handle authentication. y-websocket allows you to handle auth over cookies or request parameters. y-webrtc encrypts documents using a shared password. Yjs itself is encryption/auth-agnostic. You can implement your own auth mechanism.

    The documentation https://docs.yjs.dev is improving rapidly. If you want me to talk about how companies use Yjs, you can hit me up with a sponsorship and you can pick my brains in private sessions. There is also https://discuss.yjs.dev/ where you can ask specific questions.

    1 reply
    erdzan12
    @erdzan12
    When trying to import YJS with import * as Y from 'yjs' inside of a Polymer LitElement project, an error occurs in the browser saying Uncaught SyntaxError: The requested module '../isomorphic.js/iso.js' does not provide an export named 'default'. Is polymer not supported by default? I tried the solution of @istvank by starting the es-dev-server with --node-resolve but this does not seem to help.
    Gabriel Rogan
    @gaberogan

    Hey @dmonad, I'm trying to get YJS working with AWS API Gateway Websockets with a Node.js Lambda since lamba supports websockets as of 2018 and it seems like the most scalable solution for my CMS use-case. Do you have any advice as to what it would take to get this working?

    I'm thinking I'll basically need to update y-websockets-server by pulling out socket.io and replacing it with the AWS stuff, and then moving local state such as 'rooms' to redis (AWS Redis global datastore) to keep the lambdas stateless. Also if you think this is a bad idea, I would appreciate any advice there as well.

    Gabriel Rogan
    @gaberogan
    Edit: y-websockets-server seems outdated and not what I want. I just realized y-websocket has a server implementation. So I'm thinking I'll write a light aws wrapper for native websockets and pull out local state like docs.
    Kevin Jahns
    @dmonad
    Hi @erdzan12 The issue was discussed here: dmonad/lib0#9 Yjs is an isomorphic module (it works in the browser and in nodejs). We sometimes want to import node-native features using commonjs. es-dev-server doesn't handle commonjs modules correctly so you get this error message.
    Kevin Jahns
    @dmonad

    @gaberogan Yjs works well together with redis. You should absolutely do that. In some intervals you probably want to call a worker process that collects the room-data and persists that to a persistent database.

    The y-websockets repository contains the current server implementation (see y-websocket/bin/server.js). I think it makes sense to use that instead of writing another wrapper. You can use the persistence hook to integrate with redis.

    1 reply
    Tom Enden
    @tomenden

    Hi, I'm looking at the option of using Yjs for a collaborative code-editor product. Just started going through the tutorials, and I'm wondering if it can fit my usecase. Some background info - The product already contains an external DB with a lot of documents (each "document" for the purpose of this message is a full fileSystem representation), and has its own Authorization logic.
    Looking into https://github.com/yjs/y-websocket it seems that the server is meant to be used "as-is" (i.e I see the recommendation is to run PORT=1234 npx y-websocket-server). What would be the right way to customize this in order to allow custom authentication? Should I copy the code from .bin/server.js and modify it or can I somehow require the server part in node? I also saw that the recommendation is to keep the data in Yjs format. Seeing as this is a mature product with A LOT of users & documents, data-migration is not an option, and the system will always need to be backwards compatible. Therefore - I would like to keep the data in its current format (in which each file content is just a simple string). What are the pitfalls to that? Would it be considered a blocker in your opinion?

    High level architecture I had in mind - A room (if I understand the terminology correctly) for each Document (=fileSystem). Allow connection to the room only to authenticated users that have the permission to enter the room. On load (first connection) - initialize the room and the Y.Doc from the data in the DB. Each change will go through the yjs server, and post-update will be persisted to the db (which works in "auto-save" mode, similar to google-doc). WDYT? Any links to relevant resources would be great, as I wasn't able to find resources for what I was looking for.

    Thanks! @dmonad Yjs looks awesome :)

    Kevin Jahns
    @dmonad
    Hi @tomenden ,
    Yjs integrates well into existing communication infrastructures and it supports any kind of database. Some databases are better suited than others (postgres is better suited to store the document in some kind of interval, while NoSQL databases are also suited to store each incremental document update - see y-leveldb) You can use y-websocket as-is or include the components you need (as I do in https://github.com/yjs/yjs-demos/tree/master/demo-server). Depending on your use-case, you might need to fork it.
    There is already a lot of information on how you can build your custom providers here: https://docs.yjs.dev/
    I'm constantly improving the documentation. If you need more help I recommend to sponsor me and select the "support contract" (see https://github.com/sponsors/dmonad). This will get you one session a week with me and will also allow me to improve the documentation further.
    1 reply
    Braden
    @AdventureBeard
    Hey @dmonad , hope you had a happy new year! In your "How Yjs works from the inside" video, you mention that someone made a visualization for the data structure of a Yjs document. Is that available anywhere? Would love to have a look at it!
    Kevin Jahns
    @dmonad
    Hi @AdventureBeard , happy new year!
    https://text-crdt-compare.surge.sh/ - The version that they used had a bug: It doesn't remove formatting markers (e.g. {bold: true}). You can walk through the data structure yourself (for Y.Text: item = y.text._start; item = item.right to iterate)
    1 reply
    Braden
    @AdventureBeard
    Wow, this is fantastic! Thanks!
    Jeremy Dombrowski
    @meatflavourdev
    Is there a demonstration or library that integrates React with Yjs, something like https://github.com/relm-us/svelt-yjs for Svelte-- Might save me some time 🤷‍♂️
    4 replies
    Coder of Salvation / Leon van Kammen
    @coderofsalvation
    hi all, is there a demo for a collaborative dom-elements? I noticed the yjs/y-dom repo, but I can't find any docs or demos about it.
    Kevin Jahns
    @dmonad
    Hi @coderofsalvation I don't maintain the repository anymore. It doesn't work with the current version of Yjs.
    It was a fun experiment, but I recommend to use the editor bindings instead.
    Coder of Salvation / Leon van Kammen
    @coderofsalvation
    ah ok thanks