Hey @Yuyz0112, first of all congratulations for the great app you've made here! I was super curious and started doing some small tests and it works great and easy to get going. The one thing I have concerns about that I wanted to ask if you know is: Just by sitting and scrolling a bit and clicking on a single page for 30-50 seconds the recorded events were 2.5+mb's which could easily result in 100+ mb's per visitor and easily go into gigs of data. Just wanted to ask, do you know any way of minimizing or compressing the data and the best way to store it so that it does not take this much space?
Hi @grohsfabian_twitter
Is the situation of “100+ mb per user” actually happens? Or just an estimation based on the testing?
rrweb use an incremental op-log data structure, which contains a relative big full snapshot and many small incremental op-logs. As the time of recording increase, the size of events should increase slowly.
But there are also some ways to increase the size of events, such as:
- Disable the inline CSS option if you are sure the remote stylesheet will not change.
- Block some DOM which log many events, like animation elements.
- Use some compression algorithm like gzip to compress the data.
It would be helpful if you can share your events for a further analyze.
Thank you for such a fast response on this! It did not actually happen, it was just a calculation based on testing. I was considering the fact that 1 page could generate 2+mbs of data for each page and considering that a user can visit multiple pages in one or more sessions, then each visitor COULD have for just one session a pretty large amount of data if they navigate at least a few pages. Hope you understand what I am referring to. I was looking into the event where the body html /css is saved, to be somehow compressed down so it can be saved compressed in a database or a file for example and when needed to be decompressed
But again, thank you for your quick response on this, right now I was looking at gzip to compress the data and this might be an ideal solution but yeah, just testing and playing with it as of now ^_^
@Yuyz0112 I recorded events and send it to our server for every 10 seconds. We will store 10 seconds events as one record in our db. While replaying, we will get all events related to one session(we get events in asc order), we will concat events and give those events to rrplayer. It works fine for 30 seconds of events. For more than 1m events, I am getting this error https://gist.github.com/aaqibjavith/abe2920dded20cee094ef60b13eab618 Could you please help me to resolve this issue.
Hey there @Yuyz0112
After some more testing I see some more problems / things that can be improved and I hope I do not intrude when mentioning them:
- rrweb-player does auto play in some cases even though autoplay is false
- rrweb-player is not responsive ( not a big deal )
- Similar to @aaqibjavith I get those errors from time to time and I am sending the events to the database every few seconds + on click + on form submissions, I would have to check this more in depth to see when is happening more specifically
- The most annoying one is when trying to replay, the player with the iframe is making the browser show the "load" sign / animation infinitely in my case and you can see this happening on rrweb.io that after the page has been loaded fully, it still shows the loading icon and if you look on chrome, it shows "Waiting for github.io" and its coming from the iframe for the replayer
Hello,
I tried to install rrweb. I can record it and I can replay the events (no errors).
But the screen of the replay is just empty. The content of the homepage, where I recorded the events, is not showing.
My events look like this:{"type":3,"data":{"source":1,"positions":[{"x":984,"y":720,"id":528,"timeOffset":0}]},"timestamp":1571996501626}
{"type":3,"data":{"source":3,"id":1,"x":0,"y":1},"timestamp":1571996501626}
{"type":3,"data":{"source":0,"texts":[],"attributes":[],"removes":[{"parentId":457,"id":461}],"adds":[]},"timestamp":1571996501727}
{"type":3,"data":{"source":3,"id":1,"x":0,"y":300},"timestamp":1571996501727}
@Yuyz0112 Do you have any idea of how to draw heatmap based on events data?
rrweb.js:907 Blocked script execution in 'http://localhost:8100/' because the document's frame is sandboxed and the 'allow-scripts' permission is not set. where i should use iframe are there any good docs for using rrweb ?
I would like to monitor a web application. The initial state can be different based on login time and user. Do I correctly assume that I have to start by taking a snapshot and then start recording the events ? Later I would rebuild the snapshot and re-run the events ? I am not yet sure how the snapshotting plays into this since it is not referred to in the intial rrweb guide
Hi, rrweb is very cool, but the only problem I currently encounter is the events size. I store my rrweb events in MongoDB, and my documents reach about 400 KB in size, and I have about 100 sessions per day, which will start taking too much disk data very soon. So I am asking: did anyone succeed in compressing them? (preferably in MongoDB context)
Okay, so I came with that solution: recorded events are sent to server in "slices" (basically setInterval on client side). Each slice is then compressed with zlib, and written to database. When I need to replay the session, each slice is decompressed, and they are concatenated into single array
But I don't know whether it's best solution or not
I have some plan of optimizing the size of events which was described in this issue:rrweb-io/rrweb#151
Since I'm having a lot of company works in this month, the development may start in weeks later. Any feedback/feature-requests are welcomed in the process.
Since I'm having a lot of company works in this month, the development may start in weeks later. Any feedback/feature-requests are welcomed in the process.
Appreciate your work -- i look forward to being able to contribute back :-)
For my case i need to store only click events
@PJujunuri_twitter Maybe you can start with the internal design docs: https://github.com/rrweb-io/rrweb#internal-design
Hey @Yuyz0112 I just tested out the following simple code https://pastebin.com/Vd3VBv1H which should stop emitting after getting the first 2 events but it simply keeps on going. Is there anything that I am missing on how it should behave?
@Yuyz0112 whats your process for new versions? I'm hoping to test out the style sheet stuff in our prototype, but I couldn't figure out a clean way to get an npm dep to run from the git repo (at least not without changing our compilation processes)
coudl even just be a beta version if its easy
i can also take a stab at patching the insertRule observer to be setTimeout, tho it looks like its technically not breaking anything at the moment (just wasted data)
@dcramer just released 0.7.28
@Yuyz0112 Thank you for the answer! Also, how would I go about to simply create a snapshot of the full page only? Without any extra events. Currently, I have to do this hack that I sent you via pastebin to stop on the second event and send that snapshot. Hope you understand what I mean. Looking forward for new updates when you got the time, what you did is really nice, but has some bugs and missing "features". Again, great job!