These are chat archives for anacrolix/torrent

9th
Jan 2018
Matt Joiner
@anacrolix
Jan 09 2018 03:56
woops :)
Matt Joiner
@anacrolix
Jan 09 2018 04:07
oh you may have blocked on calls to boltdb,locking up the client?
or locking up storage
Denis
@elgatito
Jan 09 2018 05:21
was using default storage with default BoltPieceCompletion
that maybe platform/filesystem specific
as i've said, when I did benchmarked caching it could take 0.5 seconds to do a simple Put
Denis
@elgatito
Jan 09 2018 05:27
will have to find a way to link PieceCompletion to torrent, so it can serialize the whole torrent
Denis
@elgatito
Jan 09 2018 06:28
got a request from user, there is no way to change ListenAddr on the fly? It's needed when we stick to interface and the IP has changed later
Matt Joiner
@anacrolix
Jan 09 2018 06:53
Why not use 0.0.0.0 or [::] ?
Denis
@elgatito
Jan 09 2018 06:53
people use for vpn interfaces
Matt Joiner
@anacrolix
Jan 09 2018 06:53
Oh
Denis
@elgatito
Jan 09 2018 06:53
i've seen axet was doing manual mappings and listenaddr adjustments
Matt Joiner
@anacrolix
Jan 09 2018 06:54
Can we bind directly to the interface regardless of address?
I know it's possible with Linux, but don't know if it's exposed in Go
Denis
@elgatito
Jan 09 2018 07:03
looks ListenTCP needs a TCPAddr, which is describing the IP
it's fine to detect IP change from outside of the library
would be great to redefine the listeners
Denis
@elgatito
Jan 09 2018 09:38
okay, i've got a boltdb backed PieceCompletion
pieces are syncronized each 5 seconds. msgpack to pack them, saving to the whole torrent
takes 10-15ms to restore from boltdb, up to 50ms to store them (that is with bolt Put() being done, which is somehow too slow on my laptop)
looks good, but It needs Torrent.Attach()/Torrent.Detach() to be done to work
Denis
@elgatito
Jan 09 2018 09:55
so had to override the interface
Matt Joiner
@anacrolix
Jan 09 2018 11:20
what are you using to time stuff? just logging calls? what are you serializing? various properties of a Torrent?
i've just noticed that this test runs super slow on Linux: go test github.com/anacrolix/torrent -run TestClientTransferVarious
it's very fast on osx... something interesting going on
if anyone has windoze, maybe they could time it
Denis
@elgatito
Jan 09 2018 11:22
begin := time.Now()
defer func() {
    fmt.Printf("Dyration %s\n", time.Now().Sub(begin))
}()
serializing Pieces status for the whole torrent at once, [] of Index+Hash+Completion
Matt Joiner
@anacrolix
Jan 09 2018 11:23
ok
because bolt isn't working for you?
Denis
@elgatito
Jan 09 2018 11:24
it's slowing down download, and making Set() in goroutine is not an option, because then Get() is not syncronized
Matt Joiner
@anacrolix
Jan 09 2018 11:25
i really don't like bolt
Denis
@elgatito
Jan 09 2018 11:25
heh. what is the option? )
Matt Joiner
@anacrolix
Jan 09 2018 11:25
sqlite!
i think i switched from that because people didn't want cgo
i suspect a bolt option might fix this problem
i also think it might be why the test is slow on linux
something is different to my mac
i don't use bolt in production, so this never is an issue for me
Denis
@elgatito
Jan 09 2018 11:26
elgatito@HP:~/go/src/github.com/anacrolix/torrent$ go test github.com/anacrolix/torrent -run TestClientTransferVarious
ok      github.com/anacrolix/torrent    93.960s
my laptop is weak, not sure how bad is that
Matt Joiner
@anacrolix
Jan 09 2018 11:27
ok, are you on windoze, linux or mac?
i will bet ur not on mac
Denis
@elgatito
Jan 09 2018 11:27
linux
Matt Joiner
@anacrolix
Jan 09 2018 11:27
excellent
Denis
@elgatito
Jan 09 2018 11:27
ubuntu
people on windows and android were reporting bad speed using default storage
it's probably only mac running fast :)
Matt Joiner
@anacrolix
Jan 09 2018 11:30
it's more likely that mac takes a shortcut
that the others are doing properly
but since i develop on mac i've never noticed
Matt Joiner
@anacrolix
Jan 09 2018 11:36
just to check, you're only using bolt for piece completion, not for data storage right?
Denis
@elgatito
Jan 09 2018 11:36
right
Matt Joiner
@anacrolix
Jan 09 2018 11:36
ok thx
Denis
@elgatito
Jan 09 2018 11:36
separate database used only for storing piececompletion
Denis
@elgatito
Jan 09 2018 11:43
leveldb would not be an option since it would need cgo?
Matt Joiner
@anacrolix
Jan 09 2018 11:53
ok, i just confirmed boltdb is the problem
both piece completion and storage are super slow with bolt
i wonder why mac is so fast with it
Denis
@elgatito
Jan 09 2018 11:54
probably cheating
Denis
@elgatito
Jan 09 2018 12:06
github.com/syndtr/goleveldb - this looks good?
Matt Joiner
@anacrolix
Jan 09 2018 12:06
i've enabled nosync for piece completion and storage
it's ridiculously faster
Denis
@elgatito
Jan 09 2018 12:07
and Get will be syncronized?
Matt Joiner
@anacrolix
Jan 09 2018 12:07
in both cases the data is recoverable from the network so i don't think we need to be concerned about data loss
Denis
@elgatito
Jan 09 2018 12:08
if you push the change - I will test on my side
Matt Joiner
@anacrolix
Jan 09 2018 12:08
cheers
i have anotehr big change coming for linux, i've implemented recvmmsg in go-libutp
because my server hits a cpu bottleneck it's reading so fast from the network
i can get 60-80MB/s from the network
before the change
Denis
@elgatito
Jan 09 2018 12:09
my builds always use cgo, so we are probably always using c libutp?
Matt Joiner
@anacrolix
Jan 09 2018 12:09
i hope so, i abandoned the pure Go version
it was awful and the spec sux
Denis
@elgatito
Jan 09 2018 12:12
elgatito@HP:~/go/src/github.com/anacrolix/torrent$ go test github.com/anacrolix/torrent -run TestClientTransferVarious
ok      github.com/anacrolix/torrent    23.229s
4x gain
Matt Joiner
@anacrolix
Jan 09 2018 12:13
yeah and there's lots of other overhead in that test. the real gain on the network will be much more
does this help #227?
i wonder if it's mainly your linux users that complain about speeds?
Denis
@elgatito
Jan 09 2018 12:16
hope it'll help. it was reported by both windows and linux (including android) users
I will make a new release and ask for a re-test
Matt Joiner
@anacrolix
Jan 09 2018 12:23
anacrolix/go-libutp@6265f25
also anacrolix/torrent@0921889
Denis
@elgatito
Jan 09 2018 12:39
thinking whether I need to move piececompletion back

getting lots of messages:

1 <nil>
1
1 <nil>
1
4 <nil>
4
4 <nil>

is it from latest commits?

Denis
@elgatito
Jan 09 2018 12:48
func (me *Conn) RecvMsgs(ms []Message) (n int, err error) {
    defer func() {
        log.Println(n, err)
    }()
Matt Joiner
@anacrolix
Jan 09 2018 12:53
woops!
sorry
i just had another project start up again
got distracted
Matt Joiner
@anacrolix
Jan 09 2018 13:02
i remove all those debug logs thx
Matt Joiner
@anacrolix
Jan 09 2018 13:07
there's a fix for Windows too for mmsg
Denis
@elgatito
Jan 09 2018 13:40
looks good. staging a release