These are chat archives for anacrolix/torrent

11th
Jan 2018
deranjer
@deranjer
Jan 11 2018 00:47
@elgatito A little surprised that boltdb was bottlenecking, didn't think it would choke with this workload... are you looking at BadgerDB? https://github.com/dgraph-io/badger
https://github.com/dgraph-io/badger#comparisons Claims to be quicker than BoltDB, based on LSM tree

@anacrolix "// Store torrent file data in this directory unless TorrentDataOpener is
// specified.
DataDir string long:"data-dir" description:"directory to store downloaded torrent data""

I don't see anything called TorrentDataOpener in the config options... is this implemented? I'm looking at storing downloading torrents in different directories... thought I would see if that was something the library could do... If not I'm going to see if it freaks out about symlinks since the OS package does do those apparently.

Matt Joiner
@anacrolix
Jan 11 2018 01:35
@deranjer I think it's now DefaultStorage
Do you want to make an issue, or PR for it?
@elgatito The bottleneck is magnified by the fact we make separate Get calls for every single piece. Maybe we could just change the interface to []bool and reduce the overhead
I think instrumenting the number of times the raw Set/Get calls are made would determine which of those is the bigger problem (I'll bet Get), which is a natural choice to return []bool
I worked for dgraph briefly
Matt Joiner
@anacrolix
Jan 11 2018 01:40
I'd love to reenable the sqlite3 piece completion, it's still in the repo but hidden I think. Badger looks worth trying if it isn't overengineered.
deranjer
@deranjer
Jan 11 2018 01:55
@anacrolix So I would have to add all of my torrents via Infohash then? Since that seems to be the only option that accepts a "storage.ClientImpl"? When I restart my client I will need to somehow feed the storage info back into the client (if I am doing it with your library).
Also, is there any way you can Batch your Get calls? that might speed things up significantly. Badger appears to be very similiar in syntax to boltdb, so might not be too hard to implement.
Denis
@elgatito
Jan 11 2018 05:21
when we do call Piece.PieceCompletion.Set() the next millisecond Get call should receive the changed Complete value, otherwise it will unverify the piece and start download again
that was the reason i've tried the memory map, in front of boltdb, which was syncing that map
Denis
@elgatito
Jan 11 2018 05:27
the difference between badger/leveldb/bolt is somehow described here https://blog.dgraph.io/post/badger-lmdb-boltdb/
anyway, that would be easy to use any database, there are only two functions to implement
we can think about proxy layer from the library side, to have pieces' completions in memory and using Set([]bool) to write them one
Matt Joiner
@anacrolix
Jan 11 2018 11:37
@deranjer I had a look, it doesn't seem like batching will help. Something else is required. Also I don't like the TorrentSpec stuff, it sounds like it's not working for you either. Badger does look similar to Bolt, it should be pretty easy to write a storage engine for. I don't think I'll do it, I have no need for it, but copying the bolt one and modifying it would be the easiest route there.
Matt Joiner
@anacrolix
Jan 11 2018 23:46
i located the source of the slowness with piece completion
lots of speed improvements to this stuff over the last few days