Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 00:06
    lithp review_requested #1169
  • 00:06
    lithp commented #1169
  • 00:05
    pipermerriam commented #1175
  • Sep 18 23:56
    carver closed #1175
  • Sep 18 23:56
    carver opened #1175
  • Sep 18 23:51
    carver commented #1129
  • Sep 18 23:31
    pipermerriam closed #1159
  • Sep 18 23:31
    pipermerriam commented #1159
  • Sep 18 22:26
    pipermerriam synchronize #1156
  • Sep 18 22:23
    ralexstokes synchronize #1173
  • Sep 18 22:20
    ralexstokes synchronize #1173
  • Sep 18 22:13
    ralexstokes commented #1174
  • Sep 18 22:06
    ralexstokes commented #1171
  • Sep 18 22:05
    ralexstokes labeled #1171
  • Sep 18 22:05
    ralexstokes edited #1171
  • Sep 18 22:03
    ralexstokes synchronize #1173
  • Sep 18 21:56
    ralexstokes opened #1174
  • Sep 18 20:56
    ralexstokes opened #1173
  • Sep 18 20:46
    carver commented #1159
  • Sep 18 20:43
    ralexstokes commented #1172
Bryant Eisenbach
@fubuloubu
Definitely the latter
Jason Carver
@carver
Lol, then I'm the one with a lot more work to do before devcon, to make it more accessible :)
Voith Mascarenhas
@voith

Lets me try to ask you better questions. One thing is clear. I’m definitely not upto you level to understand all the gory details. So let me at least tell you my doubts.

  • Let me first explain what I've understood. So unlike other protocols beam sync starts from a checkpoint and traverses backwards which you call backfilling. Is that right?
  • I didn’t understand the attack vector how someone can pay ether and make the sync process fall behind. Can you eleaborate this a bit more.
  • apparent that the benefit of executing blocks within minutes versus executing blocks within… days

Didn’t understand this part. A block can take days to execute?

  • I’ve never experienced the 4 hours that geth claims

I’m missing context here. Is there a link for this claim

  • Incase of fast sync when it detects that its been syncing on the wrong chain, it does a binary search to find the common ancestor. How does beam sync detect that its on the wrong side of the chain? I can understand that you can verify the correct chain from earliest to latest. But I don’t fully grasp how you can correctly verify latest to earliest?

  • I just saw in one mail that etherscan is used as a checkpoint for beam sync. I didn’t get time to see the code yet. But why is trinity trusting a third party service? How can a decentralized protocol trust etherscan? Is this a hack?

@carver sorry for my stupid questions. You don’t need to answer these now. But if you think these questions are sensible enough then you can address them at devcon.

Noel Maersk
@veox
Other-topic: for the life of me, can't figure out why pip install -e .\[dev\] pulls in blspy (which is in .[eth2]).
Voith Mascarenhas
@voith
@veox You could use pipdeptree to check

But I don’t fully grasp how you can correctly verify latest to earliest?

Or do you first record the genesis block and then start from the latest block and move backwards and see if the path is right? This seems possible.

Christoph Burgdorf
@cburgdorf

beam sync starts from a checkpoint and traverses backwards which you call backfilling. Is that right?

Beam sync does only start from a checkpoint if you tell it to with --beam-from-checkpoint. The default mode relies on first syncing the chain of headers similar to fast sync.

Noel Maersk
@veox
@voith Cool, I'll take a look. (Seems like blspy is needed anyway, to run the full test suite.)
Christoph Burgdorf
@cburgdorf
@voith it's because install_requires = deps['trinity'] + deps['p2p'] + deps['eth2']
(I've recently asked myself the very same question)
Voith Mascarenhas
@voith
Thats for @veox not me :D
Noel Maersk
@veox
^_^
Voith Mascarenhas
@voith

Beam sync does only start from a checkpoint if you tell it to with --beam-from-checkpoint. The default mode relies on first syncing the chain of headers similar to fast sync.

oops wrong asumption. Now that makes sense. So its only the state sync that is different

Christoph Burgdorf
@cburgdorf

I just saw in one mail that etherscan is used as a checkpoint for beam sync. I didn’t get time to see the code yet. But why is trinity trusting a third party service? How can a decentralized protocol trust etherscan? Is this a hack?

To elaborate: By default, Trinity does not rely on any checkpoint and first syncs the entire header chain. However, syncing the header chain can also take a few hours and we wanted to give users a way to run beam sync and have a fully working clients within minutes. So you can do --beam-from-checkpoint <special-dsl> where that DSL can specify either a hash + score that you looked up yourself or it fetches one from etherscan. But again, that feature is entirely optional.

So its only the state sync that is different

Correct

And there are really cool things about that. E.g one of the natural properties of beam sync is that it asks peers about data that should usually be very fresh in other peers cache (because it asks for state nodes from recent blocks)
Voith Mascarenhas
@voith

So you can do --beam-from-checkpoint <special-dsl> where that DSL can specify either a hash + score that you looked up yourself or it fetches one from etherscan. But again, that feature is entirely optional

Thanks for clearing this. I was worried that this was part of the protocol. Ofcourse you guys are smart and won’t design something so foolish

Christoph Burgdorf
@cburgdorf
I'm not smart but my peers are :joy:
Voith Mascarenhas
@voith
Haha, modesty :D
Christoph Burgdorf
@cburgdorf
I'm counting on their cleverness to shine on me.
haha
Voith Mascarenhas
@voith
The very same reason I like contributing to this codebase. :D
Bryant Eisenbach
@fubuloubu
have a fully working clients within minutes
That's pretty sweet, I gotta say
Basically, fully trusted to start, but then you shed trust assumptions as it goes back and verified everything?
Christoph Burgdorf
@cburgdorf
Correct, it starts filling up all the missing data in the background and at one point you'll have everything fully verified. The backfill isn't yet fully implemented and it may take a while until we have fetched everything.
Bryant Eisenbach
@fubuloubu
I like this "progressive trust verification"
Jason Carver
@carver

I didn’t understand the attack vector how someone can pay ether and make the sync process fall behind. Can you eleaborate this a bit more.

Yeah, Beam Sync looks up data as it's needed by transactions in a block. If a transaction did a ton of distinct SLOADs, then Beam Sync would have to ask for a lot of data from peers. The more data it has to ask for, the more it falls behind. What does "falling behind" mean? Beam Sync starts by executing the latest block, then pulling in necessary data. If that takes ... 60 seconds, then by the time Beam Sync finishes the first block, there are four more blocks on the chain to process. You have "fallen behind" or lagged by 60s. It's very normal to be a few minutes behind the head, but if you fall behind far enough (say 30 minutes) then peers won't even serve you the data you need to execute the block and you can get "stuck". The only way out is to "pivot" over the blocks, jumping to the head again, and skipping the blocks in between. So back to the attack: if an attacker really wanted you to not process a particular block, they could load up a few blocks with a ton of SLOADs and then jam in the offending block very soon after, hoping that you will pivot over it.

apparent that the benefit of executing blocks within minutes versus executing blocks within… days

Didn’t understand this part. A block can take days to execute?

Ah, I was trying to say the time between cold startup and the first block to execute. The first block doesn't even start trying to execute for days in fast sync.

I’ve never experienced the 4 hours that geth claims

I’m missing context here. Is there a link for this claim

https://blog.ethereum.org/2019/07/10/geth-v1-9-0/ Search for "Sync time"

I really don't want to beat them over the head with this, though. I know it's not malicious. I just mean to say I haven't experienced it on my hardware, and I don't know how to replicate their conditions.

Jason Carver
@carver
@voith Yeah, it's great to get questions: I've been neck-deep in this for months, so sometimes it's hard to remember where to start :)
Voith Mascarenhas
@voith
Gotcha, thanks for the explanation. I didn’t realise that incase of sload a node would have to request for more data. Makes sense as the node wouldn’t have complete state to load the storage in the first place. Thanks for making it easy to understand.

The first block doesn’t even start trying to execute for days in fast sync.

gotcha

Voith Mascarenhas
@voith

I really don't want to beat them over the head with this, though. I know it’s not malicious.

@carver You’re not the only one. I haven’t experienced the speeds that they claim. I have 16gb Ram and ssd storage and yet it’s not as fast as I expected it to be. Ropsten took 2 days to sync with geth v1.9

sometimes it’s hard to remember where to start

:)

Jason Carver
@carver
Also, btw, there is no pivoting built-in yet. Currently, the only way to pivot is to restart trinity.
Voith Mascarenhas
@voith
I see. But its good to know you’ve already thought this through :+1:
Andrew Bezold
@AndrewBezold
I'm having difficulty finding the attributes a block object has, specifically from https://github.com/ethereum/trinity/blob/master/trinity/sync/full/chain.py#L1134. In particular, I'm trying to enhance logging by working back from X block to the last logged block and adding up the transactions and such, but without the attributes I can't figure out how to get the previous block. All I can find are abstract classes that I assume have implementations somewhere, but I can't find the implementations.
Jason Carver
@carver
The core EVM stuff is defined in https://github.com/ethereum/py-evm -- specifically, here is the StPetersburg (aka ConstantinopleFix) definition of a block: https://github.com/ethereum/py-evm/blob/dc9c03bfee31aecd100c953e8777988d8b296bd8/eth/vm/forks/petersburg/blocks.py#L16-L22
Felipe Faraggi
@faraggi
I'm compiling a list of python resources for ethereum. I have some stuff, but would gladly accept more if anyone has any ideas.
https://github.com/faraggi/ethereum-org-website/blob/python/docs/python/index.md
I'm basing it off of https://ethereum.org/java
Not necessary for it to be exhaustive, but possibly have the main ones.
thanks anyone in advance!
Noel Maersk
@veox
https://ethereum.org/java looks like tutorials; I haven't seen many web3.py-specific tutorials. https://gitter.im/ethereum/web3.py might be a better place to ask.
Voith Mascarenhas
@voith
This weeks Week in Ethereum news says:
Trinity v0.1.0-alpha.28 - get synced in an hour with BeamSync
This title is very misleading (more like clickbait). It wouldn’t have made sense to me if Jason and Christoph had not explained beam sync to me.
@carver The headers sync in just one hour ? Did you test this with discovery enabled? Or did you sync this using your own preferred node like the way you described it to me some days back?
Jason Carver
@carver
@faraggi also see http://py.ethereum.org/ for more tools
Jason Carver
@carver
@voith haha, yeah I didn't write that headline. I wouldn't summarize it that way, at least without an asterisk saying that we're being a little cheeky. I'm working on an intro to beam sync post to give more background.
Voith Mascarenhas
@voith
:D :+1:
Jason Carver
@carver

The headers sync in just one hour ?

Nope, we have to use a checkpoint to get that speed.

Did you test this with discovery enabled?

Yes, we are now at the point where we can sync against properly discovered peers on the network. You always run the risk of not finding them quickly, but you can find good-enough peers more often than not

Voith Mascarenhas
@voith

we have to use a checkpoint to get that speed.

gotcha