Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 05:36

    amark on dev

    IF email ENV CONFIG set, report… (compare)

  • 01:41

    amark on dev

    IF email ENV CONFIG set, report… (compare)

  • Jan 28 21:32
    pszabop commented #896
  • Jan 28 21:03
    pszabop edited #896
  • Jan 28 21:03
    pszabop edited #896
  • Jan 28 21:00
    pszabop opened #896
  • Jan 25 22:43
    Dletta commented #895
  • Jan 25 19:27
    bionicles edited #895
  • Jan 25 19:26
    bionicles edited #895
  • Jan 25 19:25
    bionicles edited #895
  • Jan 25 19:23
    bionicles opened #895
  • Jan 24 20:02

    amark on dev

    Merge pull request #889 from am… cache remote files, quick ugly … (compare)

  • Jan 24 19:38
    rysiekpl commented #891
  • Jan 24 19:37
    rysiekpl commented #891
  • Jan 24 18:38

    amark on master

    skip lex cursors until fixed (compare)

  • Jan 24 13:40
    mmalmi closed #760
  • Jan 24 13:40
    mmalmi closed #894
  • Jan 24 13:40

    mmalmi on master

    multicast fix tests for cursor and descending… (compare)

  • Jan 24 13:38
    mmalmi opened #894
  • Jan 24 13:29
    mmalmi synchronize #760
Fabien Bourgeois
@Yakulu
Up to the application side to filter null nodes / references, or use something like erase.js
Alisson Cavalcante Agiani
@thelinuxlich
I can't broadcast a event to all clients so they can definitely remove these empty items?
filtering null is bad because if it grows indefinitely someday you'll be iterating through a lot of dead items
Fabien Bourgeois
@Yakulu
If collection size is a concern, you could try strategies like lib/erase or lib/forget
Or custom garbage collection solution built on top of GUN, like @jabis said
Alisson Cavalcante Agiani
@thelinuxlich
what is the difference between them?
Fabien Bourgeois
@Yakulu
I guess @amark can share its experience about this and better strategies here ;)
gunchatbridge
@gunchatbridge
[D] Hans: ok this is kilmling me, cant get gun to work on webfaciton, anyone hosted there per chance ?
Fabien Bourgeois
@Yakulu
@thelinuxlich not sure about forget. erase automatically filter null nodes on subscription / iteration. It may have other use cases. I'll let someone with more experience with them than me reply here ;)
Alisson Cavalcante Agiani
@thelinuxlich
sure
Alisson Cavalcante Agiani
@thelinuxlich
well erase.js doesn't seem to do nothing to the collection of nulls
Jabis Sevon
@jabis
lib/erase will nuke the objects from the root of the graph when putting null afair
has to be included before instantiating gun (also included on all the clients) so you won't get back the merges
Jabis Sevon
@jabis
forget would be that on gun constructor options you pass {'forget':'zinggg'} then when you put({'zinggg':true}) it would delete it, if I don't remember incorrectly
too lazy to test if I remember it right or not :P
gunchatbridge
@gunchatbridge
[D] Hans: gods. finally. OK for future reference for anyone on webfaction you have to use a specific endoint for websocket and not the standard nodejs application.
gunchatbridge
@gunchatbridge
[D] marknadal: <@!114779347620986887> 👍 building your own simple CRDT sync is always awesome fun 👍
<@!280426612900757504> user.auth(pub, pass) good idea, doesn't work, tho happy to accept PR if can differentiate pub from alias safely, or just ({pub: pub}, pass maybe?
<@!203666215502217216> <@!114779347620986887> does ANY p2p system other than GUN / SEA support multi-device user login auth? Speakers at conferences keep saying "this is a hard unsolved problem" and I'm like "huh? we've been doing this since 2016" honestly I'm truly baffled at how bad all other key management is (memorize seeds, different keypairs per device, etc.) when it really doesn't seem hard at all.
Also, permissions: https://youtu.be/ZiELAFqNSLQ !
<@!114779347620986887> you don't need any of the permissions/grant/trust/etc. to do multi-device login, this works out of the box, for instance, try logging into the finished 5min interactive coding tutorial (https://gun.eco/docs/Todo-Dapp) from multiple browsers/devices & play around, still ~20 lines of code, super simple.
@AlexZan MetaMask forgot to add meeting to calendar, so will have to resync with them later, which probably won't be for at least another quarter TBH, so maybe use http://party.lol for now.
<@!355134143908151316> 🙂 thanks for helping! How goes it???
<@!470186445945176065> <@!547550778257309697> nice idea! here is a quick counter extension: https://gun.eco/docs/Counter
@rush86999 we do real world in-production scaling tests with large organizations (seen up to 50K concurrent users per second), and we have automated simulated load testing & correctness software: https://github.com/gundb/panic-server 🙂 happy to answer any Qs.
<@!643547435263852563> idk what webfaction is.
@pszabop hanging load def a bug.
[D] marknadal: <@!103660564651212800> <@!547550778257309697> <@!470186445945176065> <@!648558515463651338> figured things out? Sorry seemed like there was some confusion, very sweet to help so much. Anything I need to help with / clarify? Yes, definitely, objects should merge, not produce new/separate ones.
<@!625639230269292545> WELCOME WELCOME WELCOME TO THE COMMUNITY!!!!!!! 🤗 🤗 🤗 👏 👏 ⭐ ⚡ !!!!!! 🙂 🙂 🙂 How'd you hear about us? What are you wanting to build?
<@!103660564651212800> oh you may still need to turn localStorage off in browser? Gun({localStorage: false maybe that is the issue? How much storage left on your machine (?), some browsers like others said have other weird limits (maybe find a pure IndexedDB "fill storage" test site, and see how much it does?).
<@!636489182214357002> 👏 👏 👏
@turingmachine <@!159175656083554304> <@!565619804392652820> sync back to working (?) make sure you have /gun in the peer URL & stuff like that, also verify S3 credentials (S3 only NodeJS), do any errors log, & you can't save a primitive value (like null) to a root level node 😦 yet sowwy :(
<@!648558515463651338> love the garbage collector sweeping explanation, yes that is needed for future.
<@!159175656083554304> [native code] can't be sent remotely, seems like the logs are logging all the paramters of a function + their callbacks that are just internal JS stuff, don't worry about that stuff that isn't actual data stuff being sent between peers.
Yeah, things like erase, forget or doing Garbage Collection, are all experimental/non-existent/prototype, would love more contributions here! Definitely could use improvement. You can also just refresh your list ID, like have a list-per-day or per-hour bucket, so that way you're only iterating over new/recent keys, rather than bunch of nulled ones.
Mark Nadal
@amark
and yes... @thelinuxlich @turingmachine @brailateo WELCOME WELCOME WELCOME WELCOME TO THE COMMUNITY!!! :clap: :star: :zap: :smile_cat: :tada: How'd you hear about us? @brailateo thanks for already jumping in and trying to help too! What are you all working on building? :)
@thelinuxlich @brailateo I see Discord stole my @id to you, ^ last couple messages there was to you as well. If you have any remaining Qs on S3, radisk, nulls, etc. let me know.
Glad you all & everyone else is here!
As you can see, people <3 <3 super nice, generous, & helpful here. Life saver. :)
@go1dfish check your PMs :)
Simon Josi
@turingmachine
@amark we're building a production application for an in-store-picking application used by a food e-commerce startup.
today was the first productive run in one store with gundb.
I did disable localStorage, but I won't save any data into the IndexedDB database that actually gets created in the browser. using localStorage does work but we hit the 5M limit. also IndexedDB being async makes a lot more sense.
Simon Josi
@turingmachine
I will create a proper Github Issue for this.
gunchatbridge
@gunchatbridge
[D] Yakulu: <@!178314946201780224> Interesting counter /chain sample !
[D] bmatusiak: <@!178314946201780224> yea ({pub: pub}, pass) would work
Constantin Teodorescu
@brailateo
@amark Hello Mark, I found Gun reading some topics on Reddit, we are working since 2004 on the national animal database in Romania, hundreds of million records about owners, animals, movement, events on PostgreSQL database with Java and Go middleware. We were using CouchDB for 5 years for sync large registers of animal and events, mainly from master database to 8000 workstations. I am learning GUN to see if it fits into a mobile PWA application for veterinarians that will work also in the field where sometimes they don't have GSM signal and internet link. So the "offline first" feature is very useful. I have a lot of questions about storing large datasets in PWA, browser and RAD+indexeddb, I will try to put them in a document and send it to you! Glad to meet you, happy to discover this very-very-VERY promising software that GUN is! Best regards, Teo.
Fabien Bourgeois
@Yakulu
@brailateo Very interesting project ! Can I ask why you haven't considered PouchDB first (as it should be directly pluggable on your CouchDB database and have offline characteristics) ?
Constantin Teodorescu
@brailateo
@Yakulu 2 years ago we switched from 8000 local distributed databases to 8000 cloud databases and those large registers are available now online so we are not using anymore CouchDB. We have tested PoucDB, we are testing now a more simple way of syncing local indexeddb to PostgreSQL, we like GUN and "syncing data used by 8M+ people monthly " advertising sound very good! :-)
pszabop
@pszabop

@pszabop hanging load def a bug.

Will file. Repro is pretty straightforward. I looked at the code. It does a gun.on() without checking to see if the key exists first. FOr a subscription that's fine, but for a give me what you got API it's not

gunchatbridge
@gunchatbridge
[D] Dyno: Welcome <@232785510123307009>!!! GUN (https://github.com/amark/gun) is a P2P protocol for syncing data used by 8M+ people monthly across Internet Archive, HackerNoon, DTube (decentralized youtube), Notabug (p2p reddit), & more. Please try the 5min tutorial (https://gun.eco/docs/Todo-Dapp) to get started!
[D] ronan (wighawag):
pszabop
@pszabop
Fabien Bourgeois
@Yakulu
@brailateo Got it ! Indeed if you don't use CouchDB anymore, Pouch makes much less sense. Is Postgre data stored in relational way ? Or are there also hstore, json(b)... ?
gunchatbridge
@gunchatbridge
[D] carlosvega: <@!178314946201780224> I have no doubts i could be fun haha, but i'd rather work on top of better (tested) foundations 😛
I'm probably still doing it, but just for the lulz
And yeah, I think you're right. I ditched protocols like SSB and dat altogether bc multi device support is not there. There are ongoing efforts but it still feels like patches on top of the protocol that wasn't designed for that. Textile.io (builds on top of IPFS) seems to have some promising results, but they're in the middle of rewriting their core architecture so most of what's in their docs is basically going to need changes. Yesterday I learnt about 3Box, but I haven't had the chance to dig into it (although I know they also built their solution on top of IPFS). I really like textile cause it seems like you can get away with storing your information between your own peers (laptop, mobile phone, pc). At least that's what I understand from their docs, but then again, that's under heavy development right now. I feel like multi device support and being able to store the information solely inside your own devices is what will lead to mass adoption on decentralized systems. Tech saavy people can always spin up a server and host a peer, but most people don't understand/care about that. They just want to be able to sync their data and access it just like they currently do in web2 . Also, thanks for that video of permissions, really digging Gun so far
Constantin Teodorescu
@brailateo
@Yakulu Data in PostgreSQL is in relational tables.
gunchatbridge
@gunchatbridge
[D] thelinuxlich: <@!178314946201780224> list per day/per hour won't work in my case, it's a config object
[D] thelinuxlich: <@!178314946201780224> so there is no way to really remove the empty nodes and broadcast that to other peers so they do the same thing?
gunchatbridge
@gunchatbridge
[D] bmatusiak: <@!178314946201780224> i need statistic like you have for heroku, could you provide?
gunchatbridge
@gunchatbridge
[D] Yakulu: <@!159175656083554304> problem in decentralized system is that if a peer had the node stored locally and miss the broadcast which said to remove this node, when the peer reconnects and some other peers ask him for data, it would send back old node and the erased data will repop
[D] Yakulu: keeping it null in the graph ensures that it will be kept null, because of CRDT
[D] Yakulu: When the disconnect peer reconnects and asks for data, it will get the particular soul and its value to be null, and so can apply null value itself, ensuring coherence with other peers
Constantin Teodorescu
@brailateo
Scenario:
  • 3 browsers, A, B and C are running GUN with a super-peer (SP)
  • A is creating and saving a1 and a2 objects
  • B is creating and saving b1 and b2 objects
  • C is creating and saving c1 and c2 objects
    Questions:
    1) The superpeer (SP) has on disk saved all 6 objects (a1,a2,b1,b2,c1,c2)?
    2) If the answer to the above question is TRUE, then what storage adapter should I use in order to store and retrieve efficiently hundreds of millions of objects?
    3) If browser A try to get("c2") it will receive it from (SP) or from browser C ?
Fabien Bourgeois
@Yakulu
@brailateo
  1. Yes
  2. I'm unsure, I would say RADIX but @amark have more experience here
  3. IMHO from both, as A will ask for all connected peers, and HAM/CRDT will manage conflict and merge if there is any
Constantin Teodorescu
@brailateo
@Yakulu thanks for the answers, now for Answer 3, some more questions because in the real world, I have 8000 clients connected (I use emqtt):
  1. how many connected peers can hold browser A, I don't think it can handle 8000 connections to all other browsers
  2. if (SP) has all objects in the database, and browser C is permanently syncing with (SP), why browser A is not asking for c2 object directly from (SP)?
  3. if (SP) has all objects in the database, that means GUN is not a decentralized database, but rather a centralized database with a smart local cache for clients that are able to do a better object sync up/down to/from the central server and maybe some load distribution among peers although I can't understand how it works.
    DISCLAIMER: I apologize for my ignorance, maybe I didn't understood how data distribution is working in GUN. I have read twice the docs but there is more information for developers. The simple TODO application is way to simple to imagine how my hundred of millions of records would be store, where, how they can be retrieved, how quick, how should I design my graph of information.
Fabien Bourgeois
@Yakulu
@brailateo
  1. There is a limit to 6 peers IIRC per default. Used algorithm is explained on DAM page I guess. AXE is work in progress to strongly optimize on top of DAM (and is used massively on HackerNoon right now)
  2. Because SP is a peer with extra capabilities but a peer among others. In a decentralized system, all peers should be equal ;
  3. It is decentralized but browser can have storage limit (localstorage has 5MB hard limit, indexeddb has more) and each peer is responsible to subscribe what data it needs and so what it want to store. All peers should not have the whole dataset (but may if you want to). SP, as peer targeted by most other peers, will have larger dataset (and the whole if there is only 1 SP).
Constantin Teodorescu
@brailateo
Thanks @Yakulu . More "How GUN in the real world does this?" questions for example in the Notabug app that holds a lot of users, posts, tags, likes and comments and that is growing constantly in size:
  1. in my browser I could not find any trace of local saved information the might be related with GUN. The localStorage for https://notabug.io is empty, the indexeddb storage has something with localForage with a table "keyvaluepairs" that is empty. How Notabug is using GUN? Not in the browser?
  2. how should I deal with continuously growing the indexeddb database in the browsers in an application where new documents are permanently added? As I know , every browser has different strategies regarding the max size database can hold in indexeddb.
Fabien Bourgeois
@Yakulu
@brailateo 1. Not sure about the status of GUN usage on notabug. The author was working of alternative GUN implementation with TypeScript and crafted on the notabug.io specific needs. You could look for dweb archive.org or recent hackernoon annotations.
@brailateo 2. You can choose to only store cache through localstorage if you guess subscribed dataset should be too large for browsers, but then this will be as you said federated (if more than 1 SP) / cached architecture. You can also finely tune which data will be subscribed on browser to limit size.