datproject/discussions

discussion for the http://datproject.org, mirrored to #dat on IRC

Martin Heidegger

@martinheidegger

With the goal to encourage B to store the data locally either twice or once encrypted but decrypts it for every request.

Dat

@dat_project_twitter

emilbayes Symmetric keys are always alright :) It just depends on how you want to share it; as a shared secret or encrypted to a public key. You can never encrypt with a private key ("encrypting" with a private keys is what a signature is)

Alexander Praetorius

@serapath

@emilbayes the problem is, that using a symmetric key will give "seeders" the change to either store a single copy of the hypercore and have multiple accounts pretending to store the data multiple times ... or joining the swarm for the hypercore they store, to fetch the blocks needed to solve a challenge just in time.
Having a symmetric key, allows them to fetch the data from anywhere and encrypt it, to solve the challenge.
Having it all encrypted with the assymetric key where they do not own the private key does NOT allow them to fetch data from elsewhere just-in-time, because the only way to respond to challenges with the correct chunks is to keep the data around, because NOBODY else has it and it can't be re-created without the private key

Martin Heidegger

@martinheidegger

@serapath is my understanding correct?

Alexander Praetorius

@serapath

i'm still trying to wrap my head around it... give me some more moments :P

Dat

@dat_project_twitter

emilbayes serapath; you idea about having to prove that you have access to a block is exactly what my proof-of-storage document solves

emilbayes It sends a unique challenge that you can only solve if you actually have the data

Alexander Praetorius

@serapath

@emilbayes but you list open issues at the end, which include both, the sybil and outsourcing attack as unsolved

Dat

@dat_project_twitter

emilbayes if you don't have the data you can't solve the challenge

emilbayes What has taken Filecoin so much effort is trying to solve the same issues you listed

Alexander Praetorius

@serapath

@martinheidegger if private key Y1 is NOT under the control of B, then yes, i think what you write is correct :-)

@emilbayes i did not manage to understand how filecoin is trying to solve the problem, but also noticed, that filecoin is not yet a thing that exists... ...which makes me think they did not solve it yet

Dat

@dat_project_twitter

emilbayes serapath: https://github.com/filecoin-project/specs

Alexander Praetorius

@serapath

@emilbayes do you have experience to estimate how encrypting every chunk of an entire hypercore with a private key and decrypting every requested chunk when the data is requested with the corresponding public key would work out?

@emilbayes yes i know it, but i'm still trying to wrap my head around it. i'm slow with these things, but thx for the link.

Dat

@dat_project_twitter

emilbayes serapath it would be magnitudes faster than network I/O

Alexander Praetorius

@serapath

@emilbayes which means it would work

Dat

@dat_project_twitter

emilbayes But I don't see how you are solving sybil and outsourcing attacks

Alexander Praetorius

@serapath

i'm back in half an hour, have to jump to the shop before it closes and i try to describe it then.
The point is, that every peer which should store a hypercore has to store instead a uniquely encrypted copy of that hypercore and does not have access to that private key. The challenge tests the merkletree of that copy and without storing the data, that peer cannot re-create the data, because no access to the necessary private key

Martin Heidegger

@martinheidegger

@serapath who encrypts the data?

Dat

@dat_project_twitter

emilbayes How do you prevent peers from colluding?

Martin Heidegger

@martinheidegger

emilbayes: let me revise the understanding.

B asks A to become a responsible seed
A sends B data encrypted with private key Y1 and hands the public key Y
C requests data from B and B decrypts the received data using public key Y
D requests data but with public Key Y from B and B returns the encrypted data it originally received
D can verify that data was stored in B as it had to have the data encrypted.

This way a B2 wouldn't work as B2 would have a different private/public key and couldn't just on-the-fly create it.

Martin Heidegger

@martinheidegger

This would come with the problem that who-ever keeps the private key needs to be & stay online until B downloaded all the data.

Alexander Praetorius

@serapath

@martinheidegger the idea is that everyone who joins the network has a public/private key. If somebody wants to have their hypercore pinned/seeded a random peer is selected to retrieve the hypercore, encrypt it with their private key block by block and forward it to one randomly selected peer who announced they want to provide disk space to store hypercores.
Once that peer received the dataset, the random peer who used their private key can go offline.

Martin Heidegger

@martinheidegger

"somebody wants to have their hypercore pinned/seeded" -> A

Alexander Praetorius

@serapath

Oh interesting... I just read your summary and think that's probably even better that the peer who wants their data to be pinned encrypts it with Y1 :-) ...just one change, A can't ask B, but a random peer is selected to store the data

yes, A

Martin Heidegger

@martinheidegger

The problem with this strategy is that if the original peer A goes offline and B has to go offline for some reason another seeder couldn't be launched until A is online again.

Alexander Praetorius

@serapath

yes

that's why B who wants their data to be pinned will always cause more than one peer to store that data, so never just A, but maybe A1, A2, A3, ..., AN, where N needs to be experimented with probably its based on measurements

Martin Heidegger

@martinheidegger

I am not sure if N can ever be reasonably big enough.

Alexander Praetorius

@serapath

And actually - i said "oh interesting" above, because instead of having B encrypt the data with their private key, always a random network peer is chosen to encrypt the data... so it can also happen when B is offline

so yeah - then forget my "oh interesting" and i'd say let's stick to the random peer rather than B :-)

Martin Heidegger

@martinheidegger

The random peer is a possible idea, but uncomfortable as that peer might get a lot of work :)

Anyways: to get this you might want an extension in the hypercore protocol: https://github.com/mafintosh/hypercore-protocol#channelextensionid-buffer

https://github.com/mafintosh/hypercore-protocol#stream-message-extensions

Martin Heidegger

@martinheidegger

You add an extension "encrypted" and when the other core receives it, it returns all data calls with the encrypted values.

with the payload being the publicKey.

if the peer has data stored with the given publicKey, it can just return it. if the peer has the privateKey it can create it. If the peer has neighter then that peer's "have" becomes 0

Alexander Praetorius

@serapath

So to summarize it again:

A p2p-hashbase alternative exists with peers P1 to Pn
Seeders S1, ...Sm publish their willingness to pin hypercores
Px wants their hypercore to be pinned
The network selects q random Seeders S1' to Sq' and random peers P1' to Pq' which are not selected seeders for the given hypercore, where q is a number of seeders and their reputation is of good enough quality to ensure a quality of service for that hypercore
The data flows from Px over P1' which encrypts all hypercore blocks with their private key to S1'
- also it flows from Px over P2' which encrypts all hypercore blocks with their private key to S2'
- also it ....
- also it flows from Px over Pq' which encrypts all hypercore blocks with their private key to Sq'
The seeding starts
If Sy' goes offline a new Seeder is selected and receives data from one of the remaining Seeders which have the data over a randomly selected Peer P to ensure that new Seeder again has a uniquely encrypted copy of the hypercore

@martinheidegger @emilbayes I hope that captures it

Martin Heidegger

@martinheidegger

It does.

Alexander Praetorius

@serapath

Challenge1 In regular intervals, the network would challenge Seeders to provide one or more random chunks and the unique publicly known merkleroot of their unique copy would be used to check if the challenge response of the seeder is valid.
Challenge2 In regular intervals, a random peer would be selected to join the dat swarm for a pinned hypercore to retrieve unencrypted chunks to test if the seeder also serve the data and not just respond to "crypto challenges" and those peers reporting back to the network is also used to update scores of seeders regarding throughput and latency to figure out their "quality of service"

Will that work? ... i dont know
Is it better than whatever it is that filecoin tries to do? ...i dont know

But I'm happy for all feedback to discuss how to make this work :-)
I will definitely working on this during the next few month to make it work if it can work at all and hopefully it will work as well as possible :D

Martin Heidegger

@martinheidegger

The process seems pretty fragile. Lots of things that can go wrong :sweat:

Alexander Praetorius

@serapath

haha :-) fuck... i hope it will work

Martin Heidegger

@martinheidegger

i.e. If P1 for S1 goes down in the middle of replication ... what to do?

Alexander Praetorius

@serapath

i see. i guess start over OR continue with a different peer and their private key for remaining chunks and make a note or a list of public keys and chunks a seeder stores

technically it doesnt even matter which chunks use which private keys, just that its publicly available information i guess

Dat

@dat_project_twitter

cblgh looks like both ssb and ipfs are thinking about using base 32 (crockford's as well?)

Where communities thrive

People

Repo info

Activity