Hi @AdventureBeard , transforming a YXmlFragment should be pretty fast. From my experience, what takes a while is to load the ProseMirror document as it involves working with the dom.
Could you please share some performance profiling information? If only createNodeFromYElement takes 2.5 seconds, then there seems to be a problem.
Another problem of the Atlaskit react component is that - for whatever reason - the document is initialized several times. schema validation, rendering, decoration handling is all executed a few times when the editor is loaded. I have been struggling with Atlaskit in another project as well. Maybe you could check if you are experiencing something similar.
Offloading the ydoc⇒PM calculation is probably not an option since this would require an asynchronous calculation and you would need to load the ydoc in the worker thread as well (adding more overhead as needed).
Has anyone an idea how I can access this item id in prosemirror?
Is is possible to get something like a sync event, when yjs successfully inserted content the first time in the prosemirror state, so that I can start a scroll animation to the desired position?
(At the moment I start the scroll animation after a timeout with does not feel right. I could not find any sync event in the code...)
For the event "inserting content for the first time" you could simply observe the Yjs type and wait for it to be manipulated.
const listener = () => {
console.log('inserted content for the first time')
yxmlFragment.unobserve(listener)
}
yxmlFragment.observe(listener)There is also a synced event that is called on the provider when it first syncs with another client.
2 replies
event.transaction.local
My client shows avatars for all online peer clients, using y-protocol/awareness and y-websocket. Sometimes, the awareness message drops one of the other clients, followed immediately by adding it again. This seems to happen more often when there are many (>5) clients online, but sometimes occurs even with only two. Here's a sample trace output (logged from awareness.on('change', ...)and awareness getStates():
21:10:42.486 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
prsm.js:2716 21:10:42.488 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}
prsm.js:271 {added: Array(0), updated: Array(0), removed: Array(1)}
prsm.js:2716 21:11:14.487 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
prsm.js:2716 21:11:14.508 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}
prsm.js:271 {added: Array(0), updated: Array(0), removed: Array(1)}
prsm.js:2716 21:12:15.495 Map(5) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}}
prsm.js:271 {added: Array(0), updated: Array(1), removed: Array(0)}
prsm.js:2716 21:12:15.496 Map(6) {3772437209 => {…}, 3360666336 => {…}, 2737861673 => {…}, 3932691737 => {…}, 1856416535 => {…}, …}(e.g. at 21:11:14.487, client 182058024 was removed. 21 mSecs later, the set of clients was updated to include it again).
(NB nothing else is happening - none of the clients are broadcasting any activity at all). It is a nuisance because as a consequence the display of avatar icons flashes on each change.
Is anyone else seeing this sort of behaviour?
2 replies
new IndexeddbPersistence("my-app", doc); and now updates are basically instant...and my app works offline. Incredible.
To be explicit: You can't "feed" a Yjs document with snapshot data. You need the original content.
Snapshots (similarly to state vectors) are Uint8Arrays - binary encoded blobs of data. These blobs are not json encodable. You can send them over the via as a binary blob (most transport mechanisms support binary data), or alternatively using base64 encoding. Do not send them using JSON encoding - that would be a waste of bandwith ;)
I am unable to figure out how can i set the variables of a new Ydoc object if i have the latest doc content and all the previous snapshot data. The aim is to allow new peers to get the data from external storage when there is no other peer online to serve that data to them
You can render a specific version by changing the ySyncPlugin state. If you set version to a specific version, you render that version. If you also specify permanentUserData and prevVersion, you can render the differences. https://github.com/yjs/yjs-demos/blob/c3b1fb2a770414ff55b01cb32ac93a0282190ae6/prosemirror-versions/prosemirror-versions.js#L58
The confusing part is probably that versions are specified on the editor model and not the Yjs document.
@danny-andrews y-websocket memory leaks: yes - see yjs/y-websocket#24 The last comment in that thread suggests a resolution. I will submit a PR shortly.
Thanks @micrology
Hello, I am exploring yjs and started to use a bit with our slate editor, work out-of-box that's great! I am amazed how it works fast, I am wondering now if I can use yjs to actually make apps (like room.sh).
I am wondering if there are any architecture advices to build apps CRUD based, or open-source examples. To me, it is unclear:
- Best practice to handle relationship between models (subdocuments?)
- Best practice to handle sorting (re-ordering array indexes, building a separate array just for listing references ?)
- Best practice to handle permissions on document, if it is possible?
Thanks for any pointer !
1 reply
You can certainly use Yjs to build apps. Room.sh, and many other apps, use Yjs as a data model.
The providers handle authentication. y-websocket allows you to handle auth over cookies or request parameters. y-webrtc encrypts documents using a shared password. Yjs itself is encryption/auth-agnostic. You can implement your own auth mechanism.
The documentation https://docs.yjs.dev is improving rapidly. If you want me to talk about how companies use Yjs, you can hit me up with a sponsorship and you can pick my brains in private sessions. There is also https://discuss.yjs.dev/ where you can ask specific questions.
1 reply
import * as Y from 'yjs' inside of a Polymer LitElement project, an error occurs in the browser saying Uncaught SyntaxError: The requested module '../isomorphic.js/iso.js' does not provide an export named 'default'. Is polymer not supported by default? I tried the solution of @istvank by starting the es-dev-server with --node-resolve but this does not seem to help.
Hey @dmonad, I'm trying to get YJS working with AWS API Gateway Websockets with a Node.js Lambda since lamba supports websockets as of 2018 and it seems like the most scalable solution for my CMS use-case. Do you have any advice as to what it would take to get this working?
I'm thinking I'll basically need to update y-websockets-server by pulling out socket.io and replacing it with the AWS stuff, and then moving local state such as 'rooms' to redis (AWS Redis global datastore) to keep the lambdas stateless. Also if you think this is a bad idea, I would appreciate any advice there as well.
@gaberogan Yjs works well together with redis. You should absolutely do that. In some intervals you probably want to call a worker process that collects the room-data and persists that to a persistent database.
The y-websockets repository contains the current server implementation (see y-websocket/bin/server.js). I think it makes sense to use that instead of writing another wrapper. You can use the persistence hook to integrate with redis.
1 reply
Hi, I'm looking at the option of using Yjs for a collaborative code-editor product. Just started going through the tutorials, and I'm wondering if it can fit my usecase. Some background info - The product already contains an external DB with a lot of documents (each "document" for the purpose of this message is a full fileSystem representation), and has its own Authorization logic.
Looking into https://github.com/yjs/y-websocket it seems that the server is meant to be used "as-is" (i.e I see the recommendation is to run PORT=1234 npx y-websocket-server). What would be the right way to customize this in order to allow custom authentication? Should I copy the code from .bin/server.js and modify it or can I somehow require the server part in node? I also saw that the recommendation is to keep the data in Yjs format. Seeing as this is a mature product with A LOT of users & documents, data-migration is not an option, and the system will always need to be backwards compatible. Therefore - I would like to keep the data in its current format (in which each file content is just a simple string). What are the pitfalls to that? Would it be considered a blocker in your opinion?
High level architecture I had in mind - A room (if I understand the terminology correctly) for each Document (=fileSystem). Allow connection to the room only to authenticated users that have the permission to enter the room. On load (first connection) - initialize the room and the Y.Doc from the data in the DB. Each change will go through the yjs server, and post-update will be persisted to the db (which works in "auto-save" mode, similar to google-doc). WDYT? Any links to relevant resources would be great, as I wasn't able to find resources for what I was looking for.
Thanks! @dmonad Yjs looks awesome :)
Yjs integrates well into existing communication infrastructures and it supports any kind of database. Some databases are better suited than others (postgres is better suited to store the document in some kind of interval, while NoSQL databases are also suited to store each incremental document update - see y-leveldb) You can use y-websocket as-is or include the components you need (as I do in https://github.com/yjs/yjs-demos/tree/master/demo-server). Depending on your use-case, you might need to fork it.
There is already a lot of information on how you can build your custom providers here: https://docs.yjs.dev/
I'm constantly improving the documentation. If you need more help I recommend to sponsor me and select the "support contract" (see https://github.com/sponsors/dmonad). This will get you one session a week with me and will also allow me to improve the documentation further.
1 reply
https://text-crdt-compare.surge.sh/ - The version that they used had a bug: It doesn't remove formatting markers (e.g.
{bold: true}). You can walk through the data structure yourself (for Y.Text: item = y.text._start; item = item.right to iterate)
1 reply
4 replies
