Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 15 08:34

    romanz on v0.8.10

    (compare)

  • May 14 16:50
    RR82 commented #308
  • May 14 16:06
    RR82 commented #308
  • May 14 12:36

    romanz on master

    Bump version (compare)

  • May 14 11:34

    romanz on master

    Update release notes (compare)

  • May 14 11:06
    romanz commented #308
  • May 14 11:01

    romanz on p2p

    Link to electrumx-spesmilo.read… Add more logging in case of p2p… (compare)

  • May 10 17:45
    RR82 commented #308
  • May 08 08:14

    romanz on p2p

    Allow disabling mempool sync (compare)

  • May 08 07:09
    romanz closed #364
  • May 08 07:09
    romanz commented #364
  • May 06 18:46
    romanz commented #398
  • May 06 18:46
    romanz closed #398
  • May 06 18:46

    romanz on master

    Use more detailed error codes … Merge branch 'master-error-code… (compare)

  • May 04 19:21
    fooedge2 commented #364
  • May 04 14:31
    romanz review_request_removed #395
  • May 04 13:04
    canndrew commented #398
  • May 04 12:51
    canndrew commented #398
  • May 04 12:49
    canndrew synchronize #398
  • May 04 12:39
    romanz review_requested #398
Luke Childs
@lukechilds
I'm getting
electrs-next         | Error: failed to connect to daemon
electrs-next         |
electrs-next         | Caused by:
electrs-next         |     0: get_blockchain_info failed at 127.0.0.1:8332
electrs-next         |     1: JSON-RPC error: Hyper error: Connection refused (os error 111)
and trying to pass an address in gives me:
error: Found argument '--daemon-rpc-addr' which wasn't expected, or isn't valid in this context
Roman Zeyde
@romanz
Yes, you're right :(
I still need to port the configuration handling code from master.
Luke Childs
@lukechilds
Ah ok, that was adding the containers to my existing Docker setup where everything has it's own IP
let me try running it on the host network
btw, would it be useful to share my Docker config for testing this?
It means anyone can test electrs:next with a single command
which will pull down romanz/electrs:next and romanz/bitcoin:locations, build from the latest commit, and then run them
Roman Zeyde
@romanz

btw, would it be useful to share my Docker config for testing this?

Sounds great, please do :)

Luke Childs
@lukechilds
@romanz with --low-memory only one cpu core gets utilised, is that expected?
I guess because it's indexing one blk*.dat file at a time instead of in parallel?
Roman Zeyde
@romanz

I guess because it's indexing one blk*.dat file at a time instead of in parallel?

Yes
https://github.com/romanz/electrs/blob/edcee0ba8e2ce52ff908434261013f5c63385a68/electrs_index/src/index.rs#L323-L343

Roman Zeyde
@romanz
BTW, how much time the "low memory" indexing takes on your machine?
Luke Childs
@lukechilds
I'm recording now
running it on a Pi
Started at 12:18:00 UTC
Currently it's 15:13:36 UTC (~3 hours) and electrs DB is 4.5GB
So I guess if 30GB is the expected final DB size then it might take somewhere around 20 hours?
(30GB/4.5GB)*3 hours
Luke Childs
@lukechilds
@romanz this is my test configuration: https://github.com/lukechilds/electrs-next
Luke Childs
@lukechilds
I posted a link to it in #308 incase it's useful for anyone else wanting to test
Roman Zeyde
@romanz

I posted a link to it in #308 incase it's useful for anyone else wanting to test

Many thanks!

Luke Childs
@lukechilds

Oh also @romanz what are your thoughts on this: https://github.com/romanz/electrs/issues/308#issuecomment-716173871

The reason I ask is because we expose the user's Electrum server via a Tor hidden service for them to easily connect from remotely. It was my understanding that due to depending on Bitcoin Core RPC at query time, it would be quite easy to DoS an electrs server. So for that reason I've been advising users that it's not safe to share their electrs hidden service (unless you really trust whoever you share it with) to prevent DoS attacks.

It sounds like electrs queries bypassing RPC and instead doing direct fs reads could make this a non-issue so it would be safe for Umbrel users to share their Electrum server hidden service with friends and family. Is my assumption correct or would that still be an attack vector?

Luke Childs
@lukechilds
Also @romanz index results on a Raspberry Pi are in: https://github.com/romanz/electrs/issues/308#issuecomment-729495305

Just some stats if it's helpful for anyone, I tested this with the --low-memory flag on a Raspberry Pi 4B.

Indexed to block height 657472 with electrs efeddaa.

Indexing took ~28.5 hours.
RAM usage fluctuated between 400MB-900MB during indexing.
Index grew to ~63G before being compacted to 30GB.

Roman Zeyde
@romanz

sounds like electrs queries bypassing RPC and instead doing direct fs reads

Indeed - but there is also still the problem of "too popular" addresses (e.g. https://blockstream.info/address/bc1qjl8uwezzlech723lpnyuza0h2cdkvxvh54v3dn).
If the electrum wallet subscribes to this address, electrs will generate ~20k random file system reads, which will take a while on a (non-cached) HDD-based index.

Such addresses require separate handling to prevent DoS-ing the server.
Luke Childs
@lukechilds

Ahh ok, I understand, thanks.

If the index is sitting on a USB 3.0 SSD would that be able to handle those kinds of addresses?

Roman Zeyde
@romanz

If the index is sitting on a USB 3.0 SSD would that be able to handle those kinds of addresses?

The problem is reading the actual transactions from the disk - i.e. we need to keep the blocks' data on SSD :(

I think that it's possible to benchmark this scenario using https://github.com/axboe/fio
Roman Zeyde
@romanz
Hi all,
I have opened https://github.com/bitcoin/bitcoin#20702 to add getblocklocations RPC to Bitcoin Core.
Review and comments would be appreciated :)
dimaatmelodromru
@dimaatmelodromru
hi
is it possible to address gap limit problem with Electrs configuration?
Roman Zeyde
@romanz
@dimaatmelodromru Please elaborate.
dimaatmelodromru
@dimaatmelodromru

@romanz Electrum wallet is a recommended wallet for merchants, because it gives a (not very convenient) way to set arbitrary address gap limit, i.e., how many addresses the wallet checks for available funds before it decides that all further addresses are unused. For most wallets this value is 20. BTCPay can generate hundreds of sequential unused addresses before an invoice is actually paid.

The question is: can I tweak this parameter at Electrs server level, so that I can connect mobile Bluewallet to it and always get the correct balance (all UTXOs are collected)?

or any other Electrum client
craigraw
@craigraw
@dimaatmelodromru because electrs is a full address index, it doesn't need to support concepts like a gap limit. rather, as you suggest it is the electrum client that must specify this. i don't know how to do this on blue wallet, but with the wallet I work on, Sparrow, this can be done in the Advanced dialog on the Settings tab.
dimaatmelodromru
@dimaatmelodromru
@craigraw thanks, will check :thumbsup:
Alexz
@ph0ez_twitter
yo, have any of you ever tried to do a load testing of either electrumx or electrum server?
Alexz
@ph0ez_twitter
@romanz thanks, that was a pleasant read, btw do you know what are the benchmark stats for concurrent connections that we should aim for if we want to say "its running good"? I've clocked out on 2k with concurrencies/ disabled. And maybe you have a snippet of code that would get me started with load testing myself?
Roman Zeyde
@romanz

An update regarding current electrs optimizations:

Since the proposed approach at #308 is problematic (due to its dependence on a new bitcoind RPC), I've done some experiments with p2p-based sync (i.e. not using RPC for block retrieval from bitcoind) and it seems to work well. In addition, I've changed the index DB schema to allow the same functionality as the current master version using less storage.

Full bitcoin chain indexing takes ~4 hours while the RAM usage is less than 1GB, without any file-system access to the blk*.dat files. The resulting index takes ~28GB (as of Mar 2021) and the index queries should be as fast (and hopefully even faster) as the current master version.

I would appreciate if you could please try the latest p2p branch. Since it requires a reindex, I would recommend using the server.sh script which uses ./db2 as the default DB directory, in order to keep you previous index DB unchanged.

It should take ~1 minute to synchronize an address with ~1k transactions from scratch (with cold cache), e.g. 3Fzn4esuf5gNQEPE7ZBzWSyL4VqCiK3A2L:
[2021-03-24T18:47:13.903Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.scripthash.subscribe", "id": 21, "params": ["9a18417717a3878d2061cddc849bcd37a134532a2678bfa9be06b36ef3766170"]}
[2021-03-24T18:47:14.129Z DEBUG electrs::daemon] loading 868 blocks
[2021-03-24T18:48:00.322Z DEBUG electrs::daemon] loading 84 blocks
[2021-03-24T18:48:03.960Z DEBUG electrs::status] 1101 transactions from 951 blocks
[2021-03-24T18:48:03.962Z DEBUG electrs::server] 0: send {"id":21,"jsonrpc":"2.0","result":"39a46a556650948c745eda49624868a547730722f2eb17fd3abad64cad0e9b1c"}
...
[2021-03-24T18:48:03.963Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.scripthash.get_history", "id": 27, "params": ["9a18417717a3878d2061cddc849bcd37a134532a2678bfa9be06b36ef3766170"]}
[2021-03-24T18:48:04.073Z DEBUG electrs::server] 0: send {"id":27,"jsonrpc":"2.0","result":[{"height":640468,"tx_hash":"926dbd263bed18a3184c58d5e628f62592fcc37b835fb900be098e6995e81f68"},{"height":640540,"tx_hash":"5dc762b26ca2326b01a111d2a31e05a4cfd5dc2474950e0bd512e005>
...
[2021-03-24T18:48:04.090Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.transaction.get", "id": 28, "params": ["926dbd263bed18a3184c58d5e628f62592fcc37b835fb900be098e6995e81f68"]}
[2021-03-24T18:48:04.090Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.transaction.get", "id": 29, "params": ["5dc762b26ca2326b01a111d2a31e05a4cfd5dc2474950e0bd512e0057e7591b7"]}
[2021-03-24T18:48:04.090Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.transaction.get", "id": 30, "params": ["fdf9a3b0d91c1fc5a0dd85392ea6ee33bfeded5fe96bff81696d98255d185366"]}
[2021-03-24T18:48:04.090Z DEBUG electrs::server] 0: recv {"jsonrpc": "2.0", "method": "blockchain.transaction.get", "id": 31, "params": ["3c4038cad5fbd93b1bba52c13cc8b05d46c957507432a2a5b0b935d538c0de41"]}
...
[2021-03-24T18:48:21.029Z DEBUG electrs::server] 0: send {"id":2238,"jsonrpc":"2.0","result":{"block_height":656850,"merkle":["ffdc5c71582d1b94e4d9834e7f2607135242c0e0d40b2c93aead6917a8178489","10472d8ea4842b54ade6b840b39c182a258d9086d56eea7a7d8e1fc42c6a57b4","78abcf0d5a>
[2021-03-24T18:48:21.029Z DEBUG electrs::server] 0: send {"id":2239,"jsonrpc":"2.0","result":{"block_height":656858,"merkle":["bb14fba01fc95d8e9d509017d2917ff2b7eca1e2a46c2a199299b110d6d6378b","57ce5e67cea8cd2c7da984735b30d3c197eda85d144f9882b391d6f2a171df79","7d6cfe9cff>
[2021-03-24T18:48:21.029Z DEBUG electrs::server] 0: send {"id":2240,"jsonrpc":"2.0","result":{"block_height":657093,"merkle":["54ed26e3ab6b83f2990d96cb7ae055e69c806124e0e2776339ae7c53b9554dc2","8c70576c32c9371a50b4509b7c02f56de31d863256bfed5fee9802089b5c25d2","b835a78bb1>
[2021-03-24T18:48:21.029Z DEBUG electrs::server] 0: send {"id":2241,"jsonrpc":"2.0","result":{"block_height":657093,"merkle":["5e271494f282885cc0e6ea6ee43824a9daca683d47a8ca535150dbe53e2f78af","00c101074e73122ae42498c14984374675b033e4212572b3e22d2c6ddef70b03","6c308393bf>
Andrew Cann
@canndrew
Hi @romanz. Will there be another release of electrs based off of master? It would be useful for us at my work to get the error code fixes.
We parse error the error responses that servers return and drop the server from our list of they return and invalid JSON-RPC response, so electrs servers get dropped.
I have still have an in-progress branch for refactoring the json-rpc server btw. It just turned out to be more work than I expected so I need to find some spare time to finish it off in the next couple of weeks.
Roman Zeyde
@romanz
Thanks for the heads up!
https://crates.io/crates/electrs/0.8.10 is released :)
Andres G. Aragoneses
@knocte
@romanz git tag missing?
Andres G. Aragoneses
@knocte
we consume electrs via git :) thanks in advance