filtering null is bad because if it grows indefinitely someday you'll be iterating through a lot of dead items
Or custom garbage collection solution built on top of GUN, like @jabis said
has to be included before instantiating gun (also included on all the clients) so you won't get back the merges
too lazy to test if I remember it right or not :P
[D] marknadal: <@!114779347620986887> 👍 building your own simple CRDT sync is always awesome fun 👍
<@!280426612900757504>
<@!203666215502217216> <@!114779347620986887> does ANY p2p system other than GUN / SEA support multi-device user login auth? Speakers at conferences keep saying "this is a hard unsolved problem" and I'm like "huh? we've been doing this since 2016" honestly I'm truly baffled at how bad all other key management is (memorize seeds, different keypairs per device, etc.) when it really doesn't seem hard at all.
Also, permissions: https://youtu.be/ZiELAFqNSLQ !
<@!114779347620986887> you don't need any of the permissions/grant/trust/etc. to do multi-device login, this works out of the box, for instance, try logging into the finished 5min interactive coding tutorial (https://gun.eco/docs/Todo-Dapp) from multiple browsers/devices & play around, still ~20 lines of code, super simple.
@AlexZan MetaMask forgot to add meeting to calendar, so will have to resync with them later, which probably won't be for at least another quarter TBH, so maybe use http://party.lol for now.
<@!355134143908151316> 🙂 thanks for helping! How goes it???
<@!470186445945176065> <@!547550778257309697> nice idea! here is a quick counter extension: https://gun.eco/docs/Counter
@rush86999 we do real world in-production scaling tests with large organizations (seen up to 50K concurrent users per second), and we have automated simulated load testing & correctness software: https://github.com/gundb/panic-server 🙂 happy to answer any Qs.
<@!643547435263852563> idk what webfaction is.
@pszabop hanging load def a bug.
<@!280426612900757504>
user.auth(pub, pass) good idea, doesn't work, tho happy to accept PR if can differentiate pub from alias safely, or just ({pub: pub}, pass maybe?<@!203666215502217216> <@!114779347620986887> does ANY p2p system other than GUN / SEA support multi-device user login auth? Speakers at conferences keep saying "this is a hard unsolved problem" and I'm like "huh? we've been doing this since 2016" honestly I'm truly baffled at how bad all other key management is (memorize seeds, different keypairs per device, etc.) when it really doesn't seem hard at all.
Also, permissions: https://youtu.be/ZiELAFqNSLQ !
<@!114779347620986887> you don't need any of the permissions/grant/trust/etc. to do multi-device login, this works out of the box, for instance, try logging into the finished 5min interactive coding tutorial (https://gun.eco/docs/Todo-Dapp) from multiple browsers/devices & play around, still ~20 lines of code, super simple.
@AlexZan MetaMask forgot to add meeting to calendar, so will have to resync with them later, which probably won't be for at least another quarter TBH, so maybe use http://party.lol for now.
<@!355134143908151316> 🙂 thanks for helping! How goes it???
<@!470186445945176065> <@!547550778257309697> nice idea! here is a quick counter extension: https://gun.eco/docs/Counter
@rush86999 we do real world in-production scaling tests with large organizations (seen up to 50K concurrent users per second), and we have automated simulated load testing & correctness software: https://github.com/gundb/panic-server 🙂 happy to answer any Qs.
<@!643547435263852563> idk what webfaction is.
@pszabop hanging load def a bug.
[D] marknadal: <@!103660564651212800> <@!547550778257309697> <@!470186445945176065> <@!648558515463651338> figured things out? Sorry seemed like there was some confusion, very sweet to help so much. Anything I need to help with / clarify? Yes, definitely, objects should merge, not produce new/separate ones.
<@!625639230269292545> WELCOME WELCOME WELCOME TO THE COMMUNITY!!!!!!! 🤗 🤗 🤗 👏 👏 ⭐ ⚡ !!!!!! 🙂 🙂 🙂 How'd you hear about us? What are you wanting to build?
<@!103660564651212800> oh you may still need to turn localStorage off in browser?
<@!636489182214357002> 👏 👏 👏
@turingmachine <@!159175656083554304> <@!565619804392652820> sync back to working (?) make sure you have
<@!648558515463651338> love the garbage collector sweeping explanation, yes that is needed for future.
<@!159175656083554304>
Yeah, things like
<@!625639230269292545> WELCOME WELCOME WELCOME TO THE COMMUNITY!!!!!!! 🤗 🤗 🤗 👏 👏 ⭐ ⚡ !!!!!! 🙂 🙂 🙂 How'd you hear about us? What are you wanting to build?
<@!103660564651212800> oh you may still need to turn localStorage off in browser?
Gun({localStorage: false maybe that is the issue? How much storage left on your machine (?), some browsers like others said have other weird limits (maybe find a pure IndexedDB "fill storage" test site, and see how much it does?).<@!636489182214357002> 👏 👏 👏
@turingmachine <@!159175656083554304> <@!565619804392652820> sync back to working (?) make sure you have
/gun in the peer URL & stuff like that, also verify S3 credentials (S3 only NodeJS), do any errors log, & you can't save a primitive value (like null) to a root level node 😦 yet sowwy :(<@!648558515463651338> love the garbage collector sweeping explanation, yes that is needed for future.
<@!159175656083554304>
[native code] can't be sent remotely, seems like the logs are logging all the paramters of a function + their callbacks that are just internal JS stuff, don't worry about that stuff that isn't actual data stuff being sent between peers.Yeah, things like
erase, forget or doing Garbage Collection, are all experimental/non-existent/prototype, would love more contributions here! Definitely could use improvement. You can also just refresh your list ID, like have a list-per-day or per-hour bucket, so that way you're only iterating over new/recent keys, rather than bunch of nulled ones.
@thelinuxlich @brailateo I see Discord stole my @
Glad you all & everyone else is here!
As you can see, people <3 <3 super nice, generous, & helpful here. Life saver. :)
id to you, ^ last couple messages there was to you as well. If you have any remaining Qs on S3, radisk, nulls, etc. let me know.Glad you all & everyone else is here!
As you can see, people <3 <3 super nice, generous, & helpful here. Life saver. :)
@go1dfish check your PMs :)
today was the first productive run in one store with gundb.
I did disable localStorage, but I won't save any data into the IndexedDB database that actually gets created in the browser. using localStorage does work but we hit the 5M limit. also IndexedDB being async makes a lot more sense.
[D] bmatusiak: <@!178314946201780224> yea
({pub: pub}, pass) would work
@amark Hello Mark, I found Gun reading some topics on Reddit, we are working since 2004 on the national animal database in Romania, hundreds of million records about owners, animals, movement, events on PostgreSQL database with Java and Go middleware. We were using CouchDB for 5 years for sync large registers of animal and events, mainly from master database to 8000 workstations. I am learning GUN to see if it fits into a mobile PWA application for veterinarians that will work also in the field where sometimes they don't have GSM signal and internet link. So the "offline first" feature is very useful. I have a lot of questions about storing large datasets in PWA, browser and RAD+indexeddb, I will try to put them in a document and send it to you! Glad to meet you, happy to discover this very-very-VERY promising software that GUN is! Best regards, Teo.
@Yakulu 2 years ago we switched from 8000 local distributed databases to 8000 cloud databases and those large registers are available now online so we are not using anymore CouchDB. We have tested PoucDB, we are testing now a more simple way of syncing local indexeddb to PostgreSQL, we like GUN and "syncing data used by 8M+ people monthly " advertising sound very good! :-)
[D] Dyno: Welcome <@232785510123307009>!!! GUN (https://github.com/amark/gun) is a P2P protocol for syncing data used by 8M+ people monthly across Internet Archive, HackerNoon, DTube (decentralized youtube), Notabug (p2p reddit), & more. Please try the 5min tutorial (https://gun.eco/docs/Todo-Dapp) to get started!
[D] ronan (wighawag):
[D] carlosvega: <@!178314946201780224> I have no doubts i could be fun haha, but i'd rather work on top of better (tested) foundations 😛
I'm probably still doing it, but just for the lulz
And yeah, I think you're right. I ditched protocols like SSB and dat altogether bc multi device support is not there. There are ongoing efforts but it still feels like patches on top of the protocol that wasn't designed for that. Textile.io (builds on top of IPFS) seems to have some promising results, but they're in the middle of rewriting their core architecture so most of what's in their docs is basically going to need changes. Yesterday I learnt about 3Box, but I haven't had the chance to dig into it (although I know they also built their solution on top of IPFS). I really like textile cause it seems like you can get away with storing your information between your own peers (laptop, mobile phone, pc). At least that's what I understand from their docs, but then again, that's under heavy development right now. I feel like multi device support and being able to store the information solely inside your own devices is what will lead to mass adoption on decentralized systems. Tech saavy people can always spin up a server and host a peer, but most people don't understand/care about that. They just want to be able to sync their data and access it just like they currently do in web2 . Also, thanks for that video of permissions, really digging Gun so far
I'm probably still doing it, but just for the lulz
And yeah, I think you're right. I ditched protocols like SSB and dat altogether bc multi device support is not there. There are ongoing efforts but it still feels like patches on top of the protocol that wasn't designed for that. Textile.io (builds on top of IPFS) seems to have some promising results, but they're in the middle of rewriting their core architecture so most of what's in their docs is basically going to need changes. Yesterday I learnt about 3Box, but I haven't had the chance to dig into it (although I know they also built their solution on top of IPFS). I really like textile cause it seems like you can get away with storing your information between your own peers (laptop, mobile phone, pc). At least that's what I understand from their docs, but then again, that's under heavy development right now. I feel like multi device support and being able to store the information solely inside your own devices is what will lead to mass adoption on decentralized systems. Tech saavy people can always spin up a server and host a peer, but most people don't understand/care about that. They just want to be able to sync their data and access it just like they currently do in web2 . Also, thanks for that video of permissions, really digging Gun so far
[D] thelinuxlich: <@!178314946201780224> so there is no way to really remove the empty nodes and broadcast that to other peers so they do the same thing?
[D] Yakulu: <@!159175656083554304> problem in decentralized system is that if a peer had the node stored locally and miss the broadcast which said to remove this node, when the peer reconnects and some other peers ask him for data, it would send back old node and the erased data will repop
[D] Yakulu: keeping it null in the graph ensures that it will be kept null, because of CRDT
[D] Yakulu: When the disconnect peer reconnects and asks for data, it will get the particular soul and its value to be null, and so can apply null value itself, ensuring coherence with other peers
Scenario:
- 3 browsers, A, B and C are running GUN with a super-peer (SP)
- A is creating and saving a1 and a2 objects
- B is creating and saving b1 and b2 objects
- C is creating and saving c1 and c2 objects
Questions:
1) The superpeer (SP) has on disk saved all 6 objects (a1,a2,b1,b2,c1,c2)?
2) If the answer to the above question is TRUE, then what storage adapter should I use in order to store and retrieve efficiently hundreds of millions of objects?
3) If browser A try to get("c2") it will receive it from (SP) or from browser C ?
@Yakulu thanks for the answers, now for Answer 3, some more questions because in the real world, I have 8000 clients connected (I use emqtt):
- how many connected peers can hold browser A, I don't think it can handle 8000 connections to all other browsers
- if (SP) has all objects in the database, and browser C is permanently syncing with (SP), why browser A is not asking for c2 object directly from (SP)?
- if (SP) has all objects in the database, that means GUN is not a decentralized database, but rather a centralized database with a smart local cache for clients that are able to do a better object sync up/down to/from the central server and maybe some load distribution among peers although I can't understand how it works.
DISCLAIMER: I apologize for my ignorance, maybe I didn't understood how data distribution is working in GUN. I have read twice the docs but there is more information for developers. The simple TODO application is way to simple to imagine how my hundred of millions of records would be store, where, how they can be retrieved, how quick, how should I design my graph of information.
@brailateo
- There is a limit to 6 peers IIRC per default. Used algorithm is explained on DAM page I guess. AXE is work in progress to strongly optimize on top of DAM (and is used massively on HackerNoon right now)
- Because SP is a peer with extra capabilities but a peer among others. In a decentralized system, all peers should be equal ;
- It is decentralized but browser can have storage limit (localstorage has 5MB hard limit, indexeddb has more) and each peer is responsible to subscribe what data it needs and so what it want to store. All peers should not have the whole dataset (but may if you want to). SP, as peer targeted by most other peers, will have larger dataset (and the whole if there is only 1 SP).
Thanks @Yakulu . More "How GUN in the real world does this?" questions for example in the Notabug app that holds a lot of users, posts, tags, likes and comments and that is growing constantly in size:
- in my browser I could not find any trace of local saved information the might be related with GUN. The localStorage for https://notabug.io is empty, the indexeddb storage has something with localForage with a table "keyvaluepairs" that is empty. How Notabug is using GUN? Not in the browser?
- how should I deal with continuously growing the indexeddb database in the browsers in an application where new documents are permanently added? As I know , every browser has different strategies regarding the max size database can hold in indexeddb.
@brailateo 2. You can choose to only store cache through localstorage if you guess subscribed dataset should be too large for browsers, but then this will be as you said federated (if more than 1 SP) / cached architecture. You can also finely tune which data will be subscribed on browser to limit size.