Yjs/community

Discussions related to Shared Editing, Distributed Apps, and Yjs. Questions about Yjs should go to discuss.yjs.dev. Bug reports to the issue tracker.

trafnar

@trafnar:matrix.org

[m]

If I'm storing a ydoc as binary, then merging incoming updates into that, but one of the updates is corrupt/garbage/malicious is it possible I mess up my whole stored ydoc?

I'm worried that if I always merge the incoming data and overwrite the stored document data, I could get some broken update coming in that destroys the whole stored doc.

Jimmie Goode

@jimgoo

I was curious was the backend for wss://demos.yjs.dev looks like. Is it just a single y-websocket server with LevelDB persistence like from here https://github.com/yjs/y-websocket? It seems very reliable even with tons of demo code pointing to it. I was planning to go the y-websocket + levelDB mounted to k8s persistent volume route for my drawing app but wasn't sure how well that would scale up whenever there's more than one such pod. I'm sure the demo wss takes way more abuse that mine ever will, which is why I ask here.

Nishant Mittal

@nishantwrp

Hi, I'm running a y-webrtc signalling server that is included in the package. (https://github.com/yjs/y-webrtc). But two different machines are unable to establish webrtc connection b/w each other even when they're connected to the same signalling server. When I open two tabs in either of the machines it works i.e. two tabs in the same machine are running as expected.

Am I doing something wrong?

2 replies

alibama

@alibama

hey ya'll = running tljh with github auth = is this a fairly standard use case?

YarnBall

@yarnball

Hi folks, love this look of this project but need some help understanding some basics. I want to setup a 'middleware' (backend/sever) to verify each yjs change event (update) before writing to leveldb (or whatever gets used for sync).

I have looked at y-socket but can't find a way to do that.

Am I missing something here? Is it even possible?

Kevin Jahns

@dmonad

@jimgoo The websocket server is located here and has been running for two years ^^ https://github.com/yjs/yjs-demos/tree/main/demo-server

There's no backup. From time to time it runs out of memory and restarts - discarding all rooms that are not used anymore. It has overall ~500mb memory.

And yeah, a crazy amount of people use the demo server. Today it restarted every second. I guess somebody is bombarding it with requests again. Seems to work out though because CRDTs are pretty resilient ^^

@alibama I guess this is the wrong discussion board for this

@yarnball What does it mean "to verify" to you? You can check whether an update is parseable by decoding it. You should not filter updates depending on its content.

Ben Hohner

@benhohner

Hey there, is there any way to persist UndoManager state with the doc when persisting it, so the user can pick up where they left off in a new session with preserved editing history?

YarnBall

@yarnball

@dmonad Thanks. By verify, I mean as in checking the JSON schema of the 'data' is still valid. Also, ideally checking at the time of the request so I can "filter" the data before sending it.

Can you explain how I'd decode the data?

I've tried several methods from the docs but keep getting {} eg: https://github.com/yjs/y-websocket/blob/master/bin/utils.js#L176

My store is simple just:
{ todos: [], fragment: "xml" } (is xml the right thing here?)
Inside todo is an array of: { completed: bool, title: string }

Kevin Jahns

@dmonad

@benhohner You could disable garbage-collection (ydoc.gc = false) and store & restore the state stored in undomanager.undoStack / undomanager.redoStack after every session. There are probably quite a lot of pitfalls, but it is theoretically possible.

Kevin Jahns

@dmonad

@yarnball You should not filter updates depending on its content. That is an antipattern and prevents buggy clients from ever syncing again. You can undo changes on the backend if a client performs changes that you didn't expect. You can use the UndoManager for this.

You can't decode Yjs updates and read the changes as, for example, a diff. The updates contain CRDT operations that cannot be understood by mere humans (really scliency math stuff).

If you want to read a Yjs document, use the appropriate methods. It's always: ydoc.getText('name').toString(). The ydoc.toJSON() is deprecated and should not be used. The server must know what it reads.

YarnBall

@yarnball

UndoManager looks great.

What is the suggested pattern on returning filtered content (ie only a subset of a Yjs “room”)? Eg my data is stored inside a Y.array()

Jamalah

@jaibeee

Hey there. I'm using webrtc with the default setup and I'm receiving this error:

WebSocket connection to 'wss://y-webrtc-signaling-us.herokuapp.com/' failed: 
setupWS @ websocket.js:25
WebsocketClient @ websocket.js:122
SignalingConn @ y-webrtc.js:473
(anonymous) @ y-webrtc.js:610
setIfUndefined @ map.js:49
(anonymous) @ y-webrtc.js:610
connect @ y-webrtc.js:609
WebrtcProvider @ y-webrtc.js:595
loadDoc @ space.js?t=1664184681825:125

Is there anything I'm doing wrong or is the signaling service down?

Jamalah

@jaibeee

I'm also seeing the same error in the console on the yjs website https://yjs.dev/. The main demo subdomain is giving a heroku error https://demos.yjs.dev/.

Kevin Jahns

@dmonad

Hi @jaibeee That's definitely an issue with heroku. It's been getting very unreliable . One reason is that there is an insane amount of requests any the free instance can't handle it anymore ^^. I'm already looking to migrate to a different platform.

1 reply

@yarnball subdocuments

YarnBall

@yarnball

Thanks, saw your comment @dmonad on y-websocket.

However, the on('synced', ... does not 'fire'. You mention in your example it is 'hard to determine'. So how would I "return" the subdocument on my client?

My example: https://dpaste.org/dPPbt

YarnBall

@yarnball

Also, if my subdocuments are objects, is it best to JSON.stringify() them or use y.Set()?

EG:

const rootDoc = new Y.Doc()
const folder = rootDoc.getMap()

const subDoc = new Y.Doc()
subDoc.insert(0,  new Y.Map().set({id:123, text:'foo'}))
folder.set('123', subDoc)

Kevin Jahns

@dmonad

If you need the granularity, use a Y.Map. If not, prefer a simple JSON object (FYI Yjs accepts all kinds of JSON objects, so no need for JSON.stringify).

I'm not sure what the code on dpaste is supposed to do. The synced event only works if you use a provider. Please read the official documentation: https://docs.yjs.dev/api/subdocuments

YarnBall

@yarnball

@dmonad Thanks. I've been through the docs but it still not quite clear

For the relationship between a "subDocs" & "folder"- is it possible to get "subDocs" from multiple folders?
You say "Yjs accepts all kinds of JSON objects" but how in my example would I do that? My attempt is below, which fails saying subDoc.set is not a function. I tried using const subDoc = new Y.Doc(); subDoc.getText().insert(0, {foo:123}) but that also returns blank unless I "JSON.stringify" it

    const folder = rootDoc.getMap()
    const subDoc = new Y.Doc()
    subDoc.getMap()
    subDoc.set(data.id, data)
    folder.set(data.id, subDoc)

YarnBall

@yarnball

Here is a code example to make it more tangible https://playcode.io/971379

Kevin Jahns

@dmonad

YText only accepts string. The reason should be quite obvious. Please start with a simple example and the build up something more complex. I'm really not sure what you want to achieve - the code doesn't make it clear to me.

In Y.Map or YArray you can insert JSON objects: yarray.insert(0, [{x:1}])

This is explained with code examples here: https://docs.yjs.dev/getting-started/working-with-shared-types
There is more doc for each specific type

Jed Grant

@jedgrant

I'm developing a react firebase app locally using the firebase emulator. I'm using Yjs with Tiptap. Everything has been working great until a few minutes ago and now the console is spamming this error every couple of seconds:

WebSocket connection to 'wss://signaling.yjs.dev/' failed: Error during WebSocket handshake: Unexpected response code: 503

Any suggestions?

Erik

@erik:michelson.eu

[m]

Possibly related to this message regarding signaling server problems?
https://matrix.to/#/!bvXKFIxBJBFkoJIyBi:gitter.im/$0lifLch0N44TwnN3PrvyKeJB0bPVdmCzpm10Jdv4x8E?via=michelson.eu&via=gitter.im&via=matrix.org

Ben Hohner

@benhohner

@dmonad I think I've figured out what's going on with the high traffic, the y-webrtc-signaling-us.herokuapp.com server is failing because it seems that the y-webrtc client auto retries the server every second even if it can connect to other servers, effectively ddosing the server whenever it is down or client-server round trip latency goes above 1000ms.

Jonathan Clem

@jclem

Is there documentation on what the expectations are for server-side implementations of y-websocket? It looks like the demo server in the repo is all the documentation there really is (which is fine, it's a big project). Just trying to make sure I'm not missing anything, as I don't want to use a standalone server and want to integrate with one I already have.

I'm also interested to know what the bare minimum requirements would be for implementing a y-websocket server in a language like Go—for example, I don't need server-side storage and just want a centralized server for authenticating user access to collab sessions and proxying operations, but does building something like that still require being able to use yrs bindings?

Jonathan Clem

@jclem

Or, putting the question another way—I already have an extremely large-scale client-to-client message exchange implementation built on top of Kafka and WebSockets. Is it possible just to shuttle Yjs messages over a system like this, or is it a requirement to actually keep and store documents in-memory somewhere on the server?

Jonathan Clem

@jclem

Has anyone implemented comments in CodeMirror 6 with Y.js? It's not clear exactly what the right way is to sync the comment positions. Right now, I'm syncing them separately in a Y.Map as JSON-encoded relative positions, but if I end up accidentally applying the comment decorations before the CM6 view has apparently been updated with the document content.

MaxNoetzold

@MaxNoetzold

Hey. I have a nested Y.Map structure like that:

parentMap<Y.Map> = {
  childId1: <Y.Map> { ...childProperties },
  childId2: <Y.Map> { ...childProperties }
}

and want to use the Y.UndoManager like that:

const newUndoManager = new Y.UndoManager(parentMap);

Now. I can undo and redo events on the parentMap (e.g., add child, or remove child) but not events on the childs (i.e., property changes). As I understand the docs, this should be possible. Is there somewhere an example on how to do it?

1 reply

YarnBall

@yarnball

@dmonad Thanks for your responses, I have been trying to figure it out but I'm still stuck. This is an example adapting from y-websocket- I'm basically trying to only return a subset of the result when it goes to getYDoc

https://dpaste.org/RGOD6

Annette Bieniusa

@anne_biene_twitter

Hi, I just ran into the problem of addressing relative positions in an array and found the Y.RelativePosition thing: https://docs.yjs.dev/api/relative-positions
Can they be used with arrays? Or just with text?

As a side note, we are struggeling with implementing a collaborative spreadsheet for a student project, and finding an efficient representation with useful semantics is quite a challenge. Any ideas or help or pointers are welcome! :)

MaxNoetzold

@MaxNoetzold

Hey. I am wondering if y-websocket and y-indexeddb support Subdocs. In this forum post it says that they dont, but I would like to know if they do now, 1,5 years later

Jonathan Clem

@jclem

Reading https://discuss.yjs.dev/t/displaying-y-diffs-in-a-js-rich-text-editor-suggested-edits/1258 about suggested edits, and it seems like this should be feasible for plaintext (I'm using y-codemirror.next) without using versions/snapshots/etc just by keeping a set of YRanges for each collaborative user's changes. Then, viewing suggested changes means using mark decorations in CodeMirror to highlight them, or replace decorations when a user wants to ignore suggested edits? Maybe the difficult part here I'm ignoring is actually taking the local user edits and mapping them into YRanges?

Ah, I guess that wouldn't work for deletions—although when deleting in the editor, instead of editing text, one would add to an additional set of YRanges indicating deletions.

Jonathan Clem

@jclem

The subdocs documentation seems to imply that creating subdocs does not merge—is this correct? What happens if client A and client B concurrently try and create a subdoc with the same key? This generally seems like it's an issue with maps since their values are last-write-wins. How could one implement something where two users may concurrently create the same text document at the same time, and you want their edits to converge?

In the example on the subdocs page https://docs.yjs.dev/api/subdocuments?q=subdoc, how can client one ever be sure whether it's safe to create the document or not?

When implementing something like a file system, it seems like the root document (which of course exists implicitly on all clients) must not be a subdocument or stored in a Y.Map, because each client would have to create it and risk overwriting the root document everywhere. Instead, it seems like it would have to come from just a Y.Doc(). Maybe I'm seriously misunderstanding something, though.

Kevin Jahns

@dmonad

Hi @anne_biene_twitter,
They can be used with anything that looks like an Array (Y.Text, Y.Array, Y.XmlElement, Y.XmlFragment) ;)

@MaxNoetzold, the docs that you linked explain how you can add support to your existing provider (it's only a few lines of code).

@jclem

The subdocs documentation seems to imply that creating subdocs does not merge—is this correct? What happens if client A and client B concurrently try and create a subdoc with the same key? This generally seems like it's an issue with maps since their values are last-write-wins. How could one implement something where two users may concurrently create the same text document at the same time, and you want their edits to converge?

I'm not sure how this can happen. I expect that content is usually only created by a single user. I suspect that you are implementing a problematic pattern. In order to have two users create the same(?) content at the same time, you need coordination between these users. Yjs is coordination-free, so you'd need to implement this yourself (it would be better to avoid this problematic pattern).

I think the other questions have been posted on the discussion forum as well (if they haven't they should be). I feel the chat is not a good place to ask complicated questions. It's hard to keep track. It would be better to post to discuss.yjs.dev instead.

2 replies

Flávio Carvalho

@flaviouk

Hey all, does anyone know how to update a yXmlText embed? eg.
const yXmlText = new Y.XmlText()
yXmlText.insertEmbed(1, new Y.XmlText())
// How to get and update this nested XmlText?

Kevin Jahns

@dmonad

@flaviouk In Y(Xml)Text you currently can't get a slice of content. However, you can access the content by doing (yXmlText.toDelta()[0].insert)

Asim Aashish

@asim-aashish

just joined to say, thank you for writing yjs @dmonad

junoriosity

@junoriosity

Hi everyone,

I am currently trying to setup my own yjs backend. This backend should fulfil the following three requirements:

Depending on a provided JWT, an API call should be made for a certain document. Then the user either has read or read+write rights
For storing and syncing between the websockets there should be Redis used
The long-term storage of the documents (Quill) should be in S3. Once needed, a document will be loaded from there. Every once in a while, it will be persisted to S3. Also, once no one works on the document it will be persisted to S3 and everything document-related deleted from the backend and Redis will be removed to free resources.

Is such a thing or a similar one already implemented? If so, could you provide me with such or a nice documentation? 😀

Where communities thrive

People

Repo info

Activity