Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Kevin Jahns
    @dmonad
    Exactly, that's also what I'd recommend.
    Yousef
    @YousefED
    Hi all, I was playing around with Yjs today, and built a small library to make it easier to make your application react to changes in the document. It's a bridge between Yjs and MobX. Feedback welcome! https://github.com/YousefED/MobY and be sure to check the codesandbox demo :)
    Kevin Jahns
    @dmonad
    Sweeeet, thanks for sharing @YousefED!
    Would be great if you could also post it to https://discuss.yjs.dev/
    Yousef
    @YousefED
    Thanks, will do (after couple more iterations). One thing that would be helpful if I can more easily detect when "getters" of the Yjs types are called. For example, I now hook/patch Y.Map.get, but I should also hook toArray, toJSON, etc. I use these "hooks" to register with MobX when observers use an observable type (so I can make those run automatically after a yjs observe event). Do you know if there are lower level variable / method that would be better to intercept so I don't have to manually patch all higher level functions?
    Flávio Carvalho
    @flaviouk
    Hey, what's the tradeoff between using shared types (https://docs.yjs.dev/getting-started/working-with-shared-types#managing-multiple-collaborative-documents-in-a-shared-type) and sub documents (https://docs.yjs.dev/api/subdocuments), I got it working by using shared types quite nicely by following your example, but I'm wondering what's the tradeoff here?
    Yousef
    @YousefED
    As far as I understand, the advantage of subdocuments is that they're loaded lazily. So you're basically storing data in a different document, that will only be loaded when needed. As long as you're not worried about having extremely large documents where you load data that's not being used often, you don't need subdocuments I think.
    Flávio Carvalho
    @flaviouk
    That makes sense thanks @YousefED! I think for my use case shared types makes more sense, being able to go offline with all the information is a really nice feature, it does cost more bandwidth, but hopefully not too much, need to test this with huge documents
    Kevin Jahns
    @dmonad
    @YousefED Would it help to subscribe to an event that fires every time anything changes? I already compute a list of all types that changed.
    Yousef
    @YousefED
    @dmonad That's just observe / observeDeep right? I'm interested in the reverse; every time a read-accessor / function is called from outside the library
    Kevin Jahns
    @dmonad
    Oh got it. Unfortunately there is nothing like that.
    Henry Mao
    @calclavia
    I'm using YJS with a websocket service that has a limit on message size. If I have a long document that exceeds this size (e.g. 10kb), is there a way to easily chunk a YJS document as many small update messages? Thanks!
    Faruk Parhat
    @farukparhat

    Hey guys,
    I am trying to set up a simple synced user list using y-websockets on a node server running on Azure and a client that connects to the server.
    This works fine and dandy locally and I am able to make connections to my remote deployed Azure node server from local clients as well.
    The issue is, the userlist (awarness) is only updated on connections coming from the same device. So if I open 2 tabs in chrome and connect to the remote (Azure) node web socket server the user lists are correctly synced. However if I connect to the same web socket server with my phone at the same time, the user list is not the same between the client on my phone vs the client on my desktop (i.e different IP/devices).

    All of the clients use the same room name so I would assume they should all be getting the same awareness updates but it seems the awareness updates are only being propagated locally on clients on the same device but not across different devices.

    Here is the really simply client connection:

        const doc = new Y.Doc();
        this.wsProvider = new WebsocketProvider(
          'wss://<the-name-of-my-azure-app>.azurewebsites.net:8080',
          'my-roomname',
          doc
        );

    The server code is simple as well:

    const WebSocket = require('ws')
    const setupWSConnection = require('y-websocket/bin/utils.js').setupWSConnection
    
    const production = process.env.PRODUCTION != null
    const port = process.env.PORT || 1234
    const wss = new WebSocket.Server({ port })
    
    wss.on('connection', setupWSConnection)
    
    console.log(`Listening to http://localhost:${port} ${production ? '(production)' : ''}`)

    Is there anything I am missing here? Am I supposed to do something with the doc in such as way to ensure it is the same doc being accessed? (My assumption is, so long as the room is the same the same doc should be returned by the y-websocket-server)

    Faruk Parhat
    @farukparhat
    Apologies for the noise guys, I tried using the same websocket server as the demos (yjs) and it was working perfectly which denoted a problem with my server side not clients.
    I digged down further and found that the free tier of Azure Linux web apps lets you enable websockets but does not support them... I had to upgrade to a B2 tier and it worked perfectly after that. Curious what hosting platform are people using and any particular reasons?
    Abda Shaffan Diva
    @abdashaffan

    Hi everyone, I'm Abda and I'm new to Yjs and want to use Yjs with Ydat.
    Currently I am trying to follow examples provided from https://github.com/yjs/y-dat/blob/master/demo/node-server.cjs,but I got Error: ENOENT: no such file or directory, open '/home/abda/.local/share/dat-nodejs/78/54/78547ec5b6876100cac0087f82b965045b77f3593261e7875626f011911f30df/y-dat error, here's the minimal code that produces the error (I was just declaring the provider):

    const Y = require('yjs');
    const YDat = require('y-dat');
    
    const ydoc = new Y.Doc();
    const givenDatKey = null;
    const provider = new YDat.DatProvider(givenDatKey, ydoc);

    are there some processes that I've missed and need to do first? Because I assume that I'm not supposed to create the "missing" folder manually.

    I only have these modules as my dependencies:

        "y-dat": "0.0.1",
        "yjs": "^13.5.4"
    Kevin Jahns
    @dmonad
    @calclavia You can chunk the Yjs updates using a custom protocol (e.g. divide any update into n parts of size 10kb). Yjs (or any other CRDT) can't guarantee that you can always chunk the data structure. E.g., This would always fail when you insert content exceeding the max-message size.
    Nigel Gilbert
    @micrology
    I'm using a y-websocket server with y-leveldb persistence. I haven't been able to find out how long the leveldb database persists for - e.g. does it survive even if the y-websocket code is restarted? If the server is rebooted? I've been running it for some weeks but the earliest file date in the db directory is the lock file (dated March 19) while earliest other file is 000099.ldb, dated April 9 (it is April 12 today) which makes me think that all earlier records have been lost. If that is the case - why?
    Kevin Jahns
    @dmonad
    It should persist the data until you manually remove the database or clear the data using the y-leveldb methods
    @micrology Is there any way you can check if the data is still there. I'm not sure how meaningful the lock files are.
    @abdashaffan Have you checked out the y-dat example? I only tested it in the browser and in the dat browser. I'm not sure how well (if at all) it works in nodejs. Any patches are welcome.
    Wolfgang D
    @wdauner123_twitter
    Hi, does anyone have an idea/example on how to programmatically modify a doc in y-websockets?
    On a separate server endpoint, I need to read the doc, conditionally modify the doc contents, send the update to all clients without that update originating from one of the clients.
    Kevin Jahns
    @dmonad

    Hi @wdauner123_twitter The Y.Doc instance also exists on the y-websocket server. You could extend the Y.Doc class like I did here: https://github.com/yjs/y-websocket/blob/9d7e5eaec89050a191b301c45e484357f1c44ae8/bin/utils.js#L88

    FYI: it is generally a good idea to fork the server when you modify it. I don't have the resources to make it extensible to make it work for everyone's use-case. That's not too bad - the implementation is fairly simple

    Nigel Gilbert
    @micrology
    @dmonad About y-leveldb. It is good to know that it is supposed to persist through restarts. However, in my case, although I haven't been checking consistently, I am pretty sure that the data is being cleared (and no, I haven't removed the database nor used a delete method). The reason that it is not obvious that it is not working is that I am also using y-indexeddb, so even if the leveldb database is failing, the document is still retrieved from the local store if I have previously accessed the document. However, without leveldb, if there are no other users of the document online, and a user has not accessed it before, that user will not find the document, so I am keen to get it to work reliably. I will try to monitor what happens and report back. Meanwhile, if anyone else is using y-leveldb, it would be good to hear what your experience with it has been.
    Wolfgang D
    @wdauner123_twitter
    Thanks @dmonad I've already forked y-websocket and made other modifications, but I wondered how exactly the "get doc", "modify doc at a particular position", "send update to leveldb + all clients" steps would look like. (Example: a rest endpoint that should update the doc)
    Kevin Jahns
    @dmonad
    @micrology Where are you hosting your server? Do you have control over the filesystem?
    Wolfgang D
    @wdauner123_twitter
    Is it just getYDoc(...).getText(...).insert(...) ?
    Kevin Jahns
    @dmonad
    @wdauner123_twitter The Y.Doc automatically syncs to y-leveldb or any other provider (if you configured it). So yeah, it would be as easy as getYDoc().getText().insert(..)
    ;)
    Nigel Gilbert
    @micrology
    @dmonad Yes, I have control over the server and its filesystem
    Wolfgang D
    @wdauner123_twitter
    @dmonad Great, thanks. What would you recommend to modify a specific DeltaOp attribute in the YText? How can I find the DeltaOp and modify its attribute (given that "format" only takes the position)
    Kevin Jahns
    @dmonad
    @micrology Do you host it on a dedicated server that you own, or do you use a PaaS like google apps engine, heroku, ..
    @wdauner123_twitter You can modify the Y.Text using the methods that it provides. What exactly do you want to do?
    Wolfgang D
    @wdauner123_twitter
    @dmonad I would like to look for a DeltaOp with an attribute that has a particular app-specific ID and, if found, change the value of another attribute in that DeltaOp.
    Kevin Jahns
    @dmonad
    What do you mean with DeltaOp?
    I think you mean the Quill Delta? From my perspective this could refer to multiple things, this is why I'% asking
    Wolfgang D
    @wdauner123_twitter
    @dmonad You're right, I meant the Quill Delta. https://quilljs.com/docs/delta/#document . So for example, finding "Grey" in that example doc from the link and changing it to a lighter shade of grey :)
    Kevin Jahns
    @dmonad
    In that case, you could call ytext.toDelta() to retrieve the Quill Delta, compute the changes, and apply them in the Quill Delta format back to the ytext: ytext.applyDelta([{retain: 1, attributes: {color: 'grey'} }}]) (to make the first character bold).
    Wolfgang D
    @wdauner123_twitter

    @dmonad Great, thanks for the help!

    Another unrelated question: I want to give the persistence in y-websocket a TTL.
    My impression so far is that the best approach is a separate metadata key ("doc_exists") that has a TTL via tiny-level-ttl+subleveldown and when the key expires, it calls clearDocument for the doc.

    I'm curious if anyone has done this or a different approach.
    Nigel Gilbert
    @micrology
    @dmonad It is running on an AWS EC2 linux server, for which I am the sysadmin. At websocket = 'wss://cress.soc.surrey.ac.uk/wss';
    Kevin Jahns
    @dmonad

    @wdauner123_twitter I don't know any that have. You could store the TTL also as a metadata property alongside the database (see setMeta & getMeta).

    @micrology Thanks, in that case, I really don't know what is going on. If you are sure that documents are removed, then maybe you could add log functions to clearDocument https://github.com/yjs/y-leveldb/blob/45538cefa1f9b7b183448e89d32311331df095b8/src/y-leveldb.js#L463 and clearAll https://github.com/yjs/y-leveldb/blob/45538cefa1f9b7b183448e89d32311331df095b8/src/y-leveldb.js#L557

    The only other case when content is removed is when flushDocument is called (can be done manually, but also happens automatically). However, the algorithm is quite simple and is well tested in the testsuite

    Wolfgang D
    @wdauner123_twitter
    @dmonad Nobody cleans up the storage? Do other users of yjs-websocket keep all past stored updates forever?
    Kevin Jahns
    @dmonad
    I have little knowledge how Yjs is being used. But I assume that most users keep the Yjs document around forever. As a user, I would also expect that the document will be stored forever..
    Nigel Gilbert
    @micrology
    I've removed y-IndexedDB from my dev version, and will see what happens when I rely on levelDB on the server.
    Wolfgang D
    @wdauner123_twitter
    @dmonad My current approach for this is to store the Quill Delta via the "API callback" from the y-websockets implementation since I need information from the Quill Delta in the API anywa. If an old text needs to be retrieved much later, the y-websockets server can load it from that API. Is there a reason this is not advisable?
    Kevin Jahns
    @dmonad
    @wdauner123_twitter You can do that, but there is little advantage of doing so (except from saving some hdd space). If you persist the Yjs document forever, you can use y-indexeddb to persist / cache the document offline for faster loading times. Another advantage of using y-indexeddb is that you can basically restore content from the server even if the server lost some data ( @micrology just gave a good example). The main reason why I recommend to keep the Yjs document is because users keep running into trouble because of content duplication (an old client from a previous session rejoins a session; since the server starts fresh, the document content is duplicated).
    Wolfgang D
    @wdauner123_twitter
    @dmonad Just to clarify: do you mean to use y-indexeddb on the browser side (client) or in NodeJS (server)? If it's the browser, that would not work as long-term storage, no? In y-websocket server. I'm already using LevelDB persistence, how does y-indexeddb fit in there?
    (sorry for the many questions)
    Kevin Jahns
    @dmonad
    With y-indexeddb you duplicate the information to the clients. So even when the server loses content, you still have it on the client. In any case, the client can render the information immediately because it is available offline.
    Wolfgang D
    @wdauner123_twitter
    @dmonad Yes, that makes sense. What is the recommendation for the backend persistence? Just keep the LevelDB storage around forever (maybe GC'd) ?