substackthe tradeoff here is that it makes mutability easier but deduplication is harder
substackand you need to join fewer swarms
chamonixsubstack: So, if you have public keys of a few feeds, can you use the contents of a particlar file (that you know could be on other feeds) somehow to calculate the possible content hash of that particular file of each of the feeds (possibly "seeding" that file) if the feed was making that file available?
chamonixI mean that was an awkward sentence. I'm asking if you can check other feeds for a particular file if you know their public keys.
chamonixAnd you know they could have that EXACT file on them.
substackthere is no deduplication and the hashes will all be different
substackif you have the hash of a hyperdrive archive, you can download any files in that archive if a peer is online and cooperative
chamonixsubstack: ok, didn't know that was what deduplicatin meant. Do you know of any file sharing protocol that allows for mutability and deduplication?
substackwith ipfs last i checked you would use something like ipns to create a pointer to the root of a merkle DAG in a similar way as you might do with bittorrent using BEP44
chamonixthat's great info. I appreciate the help substack. I've heard bad things about IPFS, like it's fundamentally broken or something. I think that was someones opinion on hacker news...
substacki'm not completely sure but i think there was a recent extension announced in bittorrent that helps with deduplication but i'm not entirely sure
chamonixok, I will investigate that, thanks again.
chamonixIf I recall, the commenter critisizing IPFS said something along the lines that the design was fundamentally flawed. Something was wrong with it anyways, and the attitude was: it's used for bitcoins, and they don't care to fix it. But I could be all wrong about that...
chamonixMaybe it was IPNS, not IPFS that was broken...
okdistributeso to get the benefits of content hashing, you need the authors key, the sequence number, and the content hash -- then you can get the content. but still, deduplication across the network will still be an issue; ipfs is designed specifically for this which yes makes it really great for bitcoins & global consensus but not so great for dynamic apps
okdistributemultifeed takes a
hypercoreoption which you could replace with hyperbee, and then all the hypercores would be hyperbees instead. https://github.com/kappa-db/multifeed/blob/master/index.js#L32
chamonixokdistribute: Does that mean that, to get some range of data from the hyperbee, you would have to query every individual hyperbee for that range?
chamonixokdistribute: Is it common practice to "extract" all the data from each feed, and add it to a single hypercore?
okdistributeno, what we do in kappadb is create an index after iterating over all the feed structures
okdistributethis index could be in any on-disk or memory storage you want
okdistributelike a leveldb or whatever. but in theory you could put it in a hyperbee
okdistributeor a hypercore
okdistributethat would allow getting stored indexes from other people. you just have to trust the index you're getting from them
okdistributenot sure if that answers your question 😅 but I have to go now; good luck! interested in seeing what you're working on, if you want to share?
chamonixSo, if all the appended messages in each feed, in a multifeed, are time stamped, could you create a hyperbee database that orders each multifeed message by time stamp?
okdistributechamonix I'd recommend this talk re: timestamps. https://www.dotconferences.com/2019/12/james-long-crdts-for-mortals
okdistributechamonix in a nutshell, yes! but it gets a bit more complicated if you want to support high-reliability in ordering
chamonixAssuming every timestamp is different.
chamonixAnd when I talk about multifeeds, I'm always trying to put all the append-only data into one hyperbee.
chamonixIn my use case, all the feeds are from different authors, and they independent of each other. So if the multifeed puts these separate feeds together, can you create a single hyperbee from it?
okdistributegenderev it's in the pm2 docs https://pm2.keymetrics.io/docs/usage/startup/