Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Charanjit Singh
    @charanjit-singh_gitlab
    in Y.Doc
    Ulion
    @ulion
    @dmonad encountered a specific ydoc deadloop (take very long time to load) on the other side of y-websocket, the source side is fine, but the receive side (no other provider) will dead loop (looks like), the sync message is 600k size. for such case, what shall we do to figure out what caused this?
    Ulion
    @ulion
    截屏2021-06-25 下午3.49.12.png
    截屏2021-06-25 下午3.48.49.png
    截屏2021-06-25 下午3.47.30.png
    random debugger break got above stacks, what can I help to dig this out?
    Ulion
    @ulion
    after digging my log, it is probably caused by huge amount of repeat insert and remove element in certain place of a specific array caused by multiple clients operation dead loop. so now although the clients dead loop op is stopped, but the ydoc is still quite hard to load due to that large amount of repeat insert/delete/insert/delete ops happened before.
    Kevin Jahns
    @dmonad
    @ulion Concurrent insertions at the same position have a rather bad runtime behavior. But loading such a document after all conflicts have been resolved should still be fast. So how much time does it take to load that document?
    Ulion
    @ulion
    @dmonad it does not finish the load, at least for a few minutes.
    and cpu 100%
    I can confirm the op storm happened there, probably for 20+ minutes between 2-3 clients, so there would be large amount of insert/delete ops on the specific array.
    Kevin Jahns
    @dmonad
    Do you use any code pre-processors (e.g. babel, typescript with enabled polyfills, ..)
    ?
    Ulion
    @ulion
    yes, the final code use that. and currently it's in dev mode.
    Kevin Jahns
    @dmonad
    That could be a reason. Some polyfills for binary data are ridiculously slow. I experienced a similar issue before.
    Could you maybe share the document?
    First you should try whether the document still needs several seconds to load without polyfills.
    Ulion
    @ulion
    I think I can share the ydoc to you, via a specific websocket url.
    Kevin Jahns
    @dmonad
    Could you also download it? E.g. by calling Y.encodeStateAsUpdate(ydoc) on the loaded ydoc.
    Ulion
    @ulion
    @dmonad PM sent you the link.
    Kevin Jahns
    @dmonad

    So there is no dead loop because the process completes after 15 seconds. Apparantly, your application performs a lot of conflicts that need to be resolved. 14.5 seconds are spent in total in the integration function which iterates over all conflicts. So all seems to be in order.

    I'm not sure what you are doing exactly, but there are things that don't perform well in Yjs, or in CRDTs in general for that matter. Try to avoid creating conflicts. A few conflicts at the same position are fine. But creating many concurrent insertions at the same position should be avoided.

    Laurin Quast
    @n1ru4l
    I am working with the WebRTCProvider with document editing and have the requirement of having users join a collaboration session as read-only users. Is there an easy solution for enforcing that, did anyone already solve this issue? I assume that this is actually quite a complex problem to solve as all peers would need to know who is allowed to edit and then decline possible changes sent from the peers that are in read mode
    Ulion
    @ulion
    well, it does not load on my env in 15 seconds, it's hard to load after unload indeed, I failed to load it before websocket timeout close indeed. anyway, I found the reason caused too many conflict ops in my code and fixed the source, and deleted that ydoc and re-created it from json data. Thank you for help to dig. the conflict was created by single client itself repeat insert and delete, so it must have created huge amount conflict nodes by those op. It is reasonable since it's not normal use case in either way.
    Roman Rädle
    @raedle

    First of all, I really dig the Yjs API -- it is well-thought out!! Kudos to that!

    I am experiencing a case where an insert in a Y.Text inherits attributes from a preceeding character. Is this expected and if so, what is the reason behind it?

    Example:

    import * as Y from "yjs";
    
    const doc = new Y.Doc();
    const content = doc.getText("content");
    doc.transact(() => {
      content.insert(0, "x", { nodeType: 3, nodeName: 'span' });
      content.insert(1, "y");
    });
    
    console.log(JSON.stringify(content.toDelta(), null, 2));

    Output:

    [
      {
        "insert": "xy",
        "attributes": {
          "nodeType": 3,
          "nodeName": "span"
        }
      }
    ]

    Expected output:

    [
      {
        "insert": "x",
        "attributes": {
          "nodeType": 3,
          "nodeName": "span"
        }
      },
      {
        "insert": "y"
      },
    ]
    Roman Rädle
    @raedle

    There is also another quirk that leads to different outcomes when a document is loaded from a state update (see code below). I would expect the output in both cases to match the content delta. I am happy to file an issue on GitHub if this is not an expected behavior :)

    import * as Y from "yjs";
    
    const doc = new Y.Doc();
    const content = doc.getText("content");
    content.insert(0, "\n", { invisible: true });
    content.insert(1, "\n\n", {});
    content.format(0, 1, { nodeName: "div" });
    content.format(1, 1, { nodeName: "div" });
    content.format(2, 1, { nodeName: "div" });
    const initialState = Y.encodeStateAsUpdate(doc);
    
    function loadDoc(): Y.Doc {
      const doc = new Y.Doc();
      Y.applyUpdate(doc, initialState);
      return doc;
    }
    
    doc.transact(() => {
      content.format(1, 1, {
        nodeName: "span"
      });
    });
    
    const loadedDoc = loadDoc();
    const loadedContent = loadedDoc.getText("content");
    loadedDoc.transact(() => {
      loadedContent.format(1, 1, {
        nodeName: "span"
      });
    });
    
    console.log("content", JSON.stringify(content.toDelta(), null, 2));
    console.log("loadedContent", JSON.stringify(loadedContent.toDelta(), null, 2));

    Expected output both times:

    [
      {
        "insert": "\n",
        "attributes": {
          "nodeName": "div",
          "invisible": true
        }
      },
      {
        "insert": "\n",
        "attributes": {
          "nodeName": "span"
        }
      },
      {
        "insert": "\n",
        "attributes": {
          "nodeName": "div"
        }
      }
    ]
    Alex Chumbley
    @chumbalayaa

    Looking for a little feedback on the pattern we used to implement YJS with websockets

    1. Every time someones loads the document, we pull that document's content from a persistent store in the DB (markdown string)
    2. We create a new new Y.doc(), get the xmlFragment for our editor framework (prosemirror) and set the provider to our y-websockets server with the document id used as the room
    3. Only after the socket connection is swynced do we render our editor with that DB content. The editor takes in a markdown string
    4. Every few seconds after the editing stops, we save the editor's content to the database
    5. At the end of the React component's life, we close the socket connection

    We're seeing data loss every so often, where we pull the correct content from the database, but then on connecting to the socket server - it gets overridden with an empty document. Does the above approach make sense, or are we approaching this incorrectly? TIA

    Nick Odumo
    @nodumo
    I am having an issue with the WebRTC Provider.
    When a second user joins the WebRTC room the text in the buffer duplicates for both parties?
    For further context/clarification I am initialising the editor object with the "content" property.
    Could the provider broadcasting the initial state be causing a full duplication of the content?
    Has anyone come across this issue before?
    Nick Odumo
    @nodumo
    @dmonad , sorry to ping you but, I am rather desperate.
    addorai
    @addorai
    Hey all - we're working on a project that uses Yjs + Prosemirror. We're looking for an expert that can consult with us on a quick question (we're happy to pay at least $1K, more depending on the amount of work required). If this sounds interesting to you, feel free to reach out at aravindh.dorai@gmail.com. Thank you - much appreciated!
    Kevin Jahns
    @dmonad

    Hi all,

    I'm sorry that I haven't been so responsive anymore. As the community grew, so did the number of messages that I receive on a daily basis. It is unlikely that I will respond to your PMs. I would love to help you all, but this is too much and I'm not able to cope anymore. I will take some time off and continue answering the message boards when I'm back. I might not be able to work through the whole backlog.

    I need more people to step up and answer questions. The current workflow is not sustainable.

    Nick Odumo
    @nodumo
    No problem at all Kevin, I ended up finding a work-around for the issue that I described above. Thank you for open-sourcing this great library and the connectors
    Will Hawker
    @whawker
    @nodumo could you share your workaround?
    Alex Chumbley
    @chumbalayaa

    I also found a workaround to our issue - posting here in case anyone needs something similar

    We we pulled in our DB value and created a YDoc using prosemirrorJSONToYDoc, then looked at the Prosemirror equivalent of what was in our initial YDoc to see if it was empty. If it was, we did the following

              const stateVector1 = Y.encodeStateVector(ydoc)
              const stateVector2 = Y.encodeStateVector(dbDoc)
              const diff1 = Y.encodeStateAsUpdate(ydoc, stateVector2)
              const diff2 = Y.encodeStateAsUpdate(dbDoc, stateVector1)
              Y.applyUpdate(ydoc, diff2)
              Y.applyUpdate(dbDoc, diff1)
    kapil verma
    @kapv89
    what should be the type passed to the generic argument of Y.Map?
    kapil verma
    @kapv89
    nevermind found out
    Will Hawker
    @whawker
    thanks @chumbalayaa! What did you if it ydoc wasn't empty?
    Alex Chumbley
    @chumbalayaa
    @whawker we just loaded the ydoc data into our editor and used that as our source of truth
    Yonggoo Noh
    @ygnoh
    Is there a binding for input or textarea element?
    Namit Chadha
    @nc
    I had a question about Y.Map vs Presence for tracking something that's hard-realtime like dragging an element, what's the best data structure to use for those kind of situations.
    Eduard Ruzga
    @wonderwhy-er
    I was wondering, is there suggested strategy for cleaning up yjs doc. Have a hobby markdown/wiki where I am storing state of documents on server in sql database.
    For one often changed document with mostly links I am in situation where it contains 76 bytes of text but its base64 encoded yjs state is 65Kb and applyUpdate starts to take considerably more time.
    I was thinking about finding a condition under which I create a new doc on a server, insert text from old one, this way erase history so new connecting clients get that.
    But as far as I can say this may lead to occasional loss or duplication of text depending on how I handle case where there was client that still had state from before erasure.
    Did anyone encountered such a need? Issues with slow applyUpdate in almost empty document. Any strategies to clean things up occasionally?
    Kevin Jahns
    @dmonad
    Hey,
    so I'm back from vacation. I think it would be best if you used https://discuss.yjs.dev/ for questions for things that can't be answered in a single sentence. I'm slowly catching up to all the missed messages.
    Eduard Ruzga
    @wonderwhy-er
    Thanks, will ask there.
    Marcelo Luiz Onhate
    @onhate
    hello folks, simple question: Having a XML in string format <a>b</a> how do I create a XMLFragment with that string content?
    Marcelo Luiz Onhate
    @onhate
    or, having a XmlFragment how to I create another XmlFragment with the same content
    Marcelo Luiz Onhate
    @onhate
    I tried the following but the result is always an empty string
     const update = Y.encodeStateAsUpdate(fromDoc.doc);
     Y.applyUpdate(toDoc.doc, update);
     console.log('xml', toDoc.toJSON());
    Rhys Brett-Bowen
    @rhysbrettbowen
    Reading through the paper for YATA it mentions that it can be used for graphs. Is there an example of this? From what I could figure people usually have a map of nodes and a map of edges. With that that though, what happens if someone removes a node (and all connecting edges) in their client and another client adds an edge to that node? It seems like the new edge would still be added. This seems like it would be an issue with any piece of data that had multiple parents (as otherwise they could just be put in a map). I could imagine a garbage collector that would check and remove these as the changes came in, but is there a better way? Guessing that a graph like that (or tabular data) could be implemented as types...
    Nigel Gilbert
    @micrology
    @rhysbrettbowen There is an example of using yjs with vis-network to manage and display a graph at https://github.com/micrology/y-vis-network . This is just a skeleton example. For the full works, see https://prsm.uk and the code at https://github.com/micrology/prsm (for the interaction with yjs, see the top of https://github.com/micrology/prsm/blob/master/js/prsm.js
    Rhys Brett-Bowen
    @rhysbrettbowen
    @micrology Thanks for the links. I ran the example and can confirm that what I said is an issue. If you run 2 clients, and add 2 nodes, disconnect one, then add a link in one client and remove a node in another then reconnect you will end up with an extra edge in the edge data set. The client does not have an issue with this because the vis-network won't connect an edge where the nodes don't exist: https://github.com/visjs/vis-network/blob/master/lib/network/modules/components/Edge.js#L539. To fix this there would need to be extra logic when adding an edge to check that the node was not removed and not add it - or add it as a deleted node? I don't believe that's too bad though. For more context we've got some OT applications that do spreadsheets (which have a similar issue as a cell has a row and column - thus multiple parents) and this is handled in the transformers (because they know to look for a delete of either parent and so can remove the op). We've also got a graph application which is being done now. So the three options would be 1) leave it - not too bad as we'd just be marking them deleted anyway. It would be up to application code to make sure it deals with orphaned items, this would only be an issue if the values were large 2) change the bindings to check and fire a remove if the item is orphaned - adds extra logic to the bindings 3) have a process that garbage collects orphaned items - means a whole other process and the app will still have to deal with orphans
    Though it might be preferred to leave as is so an undo on removing the node will also bring back that edge...
    Nigel Gilbert
    @micrology
    I've just tried this and you are right - PRSM reports that the 'network is corrupt: links are connected to non-existent factors' (at least our error checking logic works!). For us this is a really edge (pardon the pun) case, as our users are typically always on line, but nevertheless, it is something we should deal with.