ethereum/tests

Metropolis Testing Guide - https://gist.github.com/Souptacular/fd197b1fac7c6d2660b0bef27a33ed40

Andrei Maiboroda

@gumb0

I'd say it's very important to execute all the previous fork rules, especially when client developers make changes in consensus-critical code (which is all the time during new fork development)

winsvega

@winsvega

the sealed test suite is available for execution. not available for changes

so not to break any of the previously created tests

Danno Ferrin

@shemnon

So would the older test suites be capped at a specific fork? i.e. when balances change EIPs are written do a new suite of tests need to be written for that fork and that fork alone?

winsvega

@winsvega

yes. will be capped at current ConstantinopleFix
all the test cases remain to be executed on new forks.
new tests could still be added for previous forks

Danno Ferrin

@shemnon

what about baseline “dose your evm function sensibly” tests, will there need to be new ones for each new fork or will some subset of the tests be updated for each fork? Or will stuff like balance checks be removed for that set of tests?

winsvega

@winsvega

all tests will be updated for each new fork. but not for previous forks. (right now it's updated for all forks with each new fork and new change)

postConditions for balance better be changed to smth more reliable and not conflicting if possible. although the fork interference might be resolved with Fortname + EIP + EIP chain params scheme where a specific test is executed only on a specific eip applied on top of fork. But this also brings up a question how a particular test case functions when many EIPs applied simultaneously

Danno Ferrin

@shemnon

I think the fork+EIP model is fine for development but I have serious doubts about the scalability for released forks.

James Hancock

@MadeofTin

The original proposal was to have fork+eip1+eip2 respect the ordering

So fork+eip2+eip1 would execute a different order of tests

rain

@rainbreak

hey, what's the story with these tests: RevertPrecompiledTouch_storage_d0g0v0|RevertPrecompiledTouch_storage_d3g0v0? (for petersburg) I can see that geth has them as known failures, but not sure why they're still in the test suite.

rain

@rainbreak

i'm guessing it's to do with special casing of ripemd, following https://github.com/ethereum/go-ethereum/pull/3341/files#diff-2433aa143ee4772026454b8abd76b9dd, but not totally sure

Andrei Maiboroda

@gumb0

@rainbreak correct, it's about that case, though I think _storage_ ones are later added cases similar to what happened on the mainnet, but with non-empty storage

geth should unmark these cases as known failures, because the tests were recently changed to reflect the result produced by geth

rain

@rainbreak

thanks. I still don't understand why the tests exist for subsequent hardforks. in the case of ConstantinopleFix, is the client expected to specifically special case ripemd?

Andrei Maiboroda

@gumb0

It should if it wants to pass the tests, but it can't happen on the mainnet

(geth has it as a special case regardless of block number, and aleth now, too)

rain

@rainbreak

ah ok, it's because the tests follow behaviour of the clients

well, apart from these two tests, hevm is now passing all of the constantinople fix GSTs :)

Martin Holst Swende

@holiman

Re the fork+eip model, I have now added more tests of 1884 and one also a test for 1344at https://github.com/holiman/IstanbulTests

and yes, it's for development -- however, the idea is to take them and turn them into Istanbul blockchain tests later

And I'm also trying to generate the same thing (minus traces and dumps) using retesteth, but haven't gotten it going quite yet

Martin Holst Swende

@holiman

@shemnon do you have the inputs to the blake tests somewhere in a more concise format? As in, only the inputs and the expected outputs?

Danno Ferrin

@shemnon

https://github.com/PegaSysEng/pantheon/blob/2168b8cfefe4ec8daf7bf4b83f0524ce29fc26ba/ethereum/core/src/test/java/tech/pegasys/pantheon/ethereum/mainnet/precompiles/BLAKE2BFPrecompileContractTest.java#L42

thats for the 20 main vectors. The rest are derivitive in the json files.

Martin Holst Swende

@holiman

Thanks!

Ah, they're from teh keep-network PR. They just nuked those, due to changes in the endian-ness

That's why I was looking for other sources of tests....

See keep-network/go-ethereum#4

Danno Ferrin

@shemnon

Makes sense. The main input was just increasing bytes. The message state hash however looks to be internal state from an iteration.

Martin Holst Swende

@holiman

After we 'align' the pantheon/geth implementation, would be good to generate some random inputs, at least as pure json input/output tests (we don't have to make blockchain tests out of every single vector)

Danno Ferrin

@shemnon

Is this the current spec? - ethereum/EIPs#2136 - I’m not seeing the f zero/non-zero or zero/one claifications, unless I missed it.

Martin Holst Swende

@holiman

hmm.., no that looks like an old version, to me

This one? https://github.com/ethereum/EIPs/pull/2129/files#diff-f7128fc1b55ed9139dcaab5ab4c91c28R51

Martin Holst Swende

@holiman

Are there any new transaction tests made for eip 2028 yet?

GuthL

@GuthL

Not as far as I know (just joined the chat). We reached out to various people to see how as champions, we can help on this front.

Martin Holst Swende

@holiman

Hm, wait, it may not be possible -- transaction tests don't have the concept of a balance... so what we need are statetests where the balance is low enough that they wouldn't work pre-istanbul, but are sufficient for 2028.

Should be fairly simple to add a couple of such tests, really

Oh no wait, transaction tests will do fine, since balance doesn't matter, what matters is the gas

Martin Holst Swende

@holiman

I continued generating Istanbul-tests with geth. If anyone wants to validate it, filled transaction tests for EIP 2028 are available at https://github.com/holiman/IstanbulTests/tree/master/TransactionTests/ttEIP2028

James Hancock

@MadeofTin

👏👏

Tomasz Kajetan Stańczak

@tkstanczak

I may load these tomorrow, no promise yet but I am just looking at testing now

Danno Ferrin

@shemnon

@holiman do you have fillers for the state tests you published?

Martin Holst Swende

@holiman

@shemnon Yes and no. I generated it using native go, here: https://github.com/holiman/go-ethereum/blob/generate_statetests/tests/statetestgenerator_test.go#L43 . However, I did also convert it to filler-form, in order to run them through the retesteth-process and get that working (see what's needed for retesteth to support base+eip form). I ran into some problems, and haven't had time to finish that

Is it possible for you to run the tests themselves and check if they seem to pass / are on correct form?

Danno Ferrin

@shemnon

I would need to hand load them and adjust the network target. It’s been on my to-do list but I haven’t been able to get that far down my list in the last few days.

However the new “from a file in a directory” chain params method may make the +EIP form a bit verbose.

Danno Ferrin

@shemnon

I adapted your GeneralStateTests into filler form for retesteth: ethereum/tests#627

let me know what you think of how I re-worked the gas tests. I think it’s easier to compare to the spec since we are getting the net gas cost of one operaton.

Danno Ferrin

@shemnon

retesteth will need this patch - shemnon/retesteth@e7eb725

Where communities thrive

People

Repo info

Activity