These are chat archives for anacrolix/torrent

25th
Jan 2018
Matt Joiner
@anacrolix
Jan 25 2018 01:46
@elgatito thank you, good catch. i'm guessing this is since the file priorities were added
@deranjer Do you mean the torrent.File.Length?
deranjer
@deranjer
Jan 25 2018 01:48
No, Torrent.Length(). I use Torrent.Length() to calculate the torrent percent complete.. If I cancel a file it obviously never downloads so my %complete will not reach 100% since all the files that WERE downloaded don't equal the full Torrent.Length()
It doesn't make sense for your library to base the torrent length off of a canceled file though
So I assume I will need to work around it.. won't be that hard
Matt Joiner
@anacrolix
Jan 25 2018 02:14
yeah it's definitely fixed
you could calculate the total expected length by summing Torrent.Files.Length where each File.Priority is not PiecePriorityNone
I don't see why a helper can't be added
hm, the only problem is that only files have priorities that stick to non-zero if set. if you set some pieces to download, those priorities will vanish when they're retrieved, so a built-in progress might be off there
i've just added very basic upnp nat support
it's just an experiment, i'm not sure i'll keep it in the main library
deranjer
@deranjer
Jan 25 2018 02:20
Yeah I plan to sum the file length of all the pieces that are not canceled and store that in the db and update any time a piece priority changes. I'll just rewrite my calculation loop to use the db value instead of the library call.
upnp is a nice feature, not sure how many people will use it for my project...
Matt Joiner
@anacrolix
Jan 25 2018 02:21
yeah nobody will use it in mine
but it's kind of a smart default for typical clients
i added a flag to turn it off for now, because it's actually not good in some situations
but it defaults to on
denis found a bug with the new file-priorities, i'm just fixing that now. it requires a bit of a rethink to how priorities work
if you have any unit tests to contribute for file-related stuff that would be huge
deranjer
@deranjer
Jan 25 2018 02:25
I hope to start writing unit tests for my code in about 2 months once I have the major features in place. I just need to do some tutorials and learning on how to write tests.
Matt Joiner
@anacrolix
Jan 25 2018 02:25
no problem
i like to run downstream tests with my own stuff, so if you have any repos i should run tests against let me know
deranjer
@deranjer
Jan 25 2018 02:27
Unfortunately currently all of my file operations are all tangled up in my Engine package. I'm debating seperating them for a more logical structure but haven't gotten around to that yet
Also, have you done any pprof work on your library, especially for Windows?
Matt Joiner
@anacrolix
Jan 25 2018 02:35
none on windows. heaps on osx and linux
performance does evolve a lot with new features tho, so if you spot something unusual, send a profile
deranjer
@deranjer
Jan 25 2018 02:39
I've just noticed it seems to gobble up memory sometimes on Windows... my app does at least, no idea if related to your library or my app. On linux much less memory usage.
Matt Joiner
@anacrolix
Jan 25 2018 02:39
what storage are you using?
deranjer
@deranjer
Jan 25 2018 02:40
boltdb
Matt Joiner
@anacrolix
Jan 25 2018 02:40
for piece/file data too?
deranjer
@deranjer
Jan 25 2018 02:40
oh yeah.. um.. is that the defautl?
im using whatever the default is
Matt Joiner
@anacrolix
Jan 25 2018 02:40
i think the default is file-based storage
if you're using bolt/mmap, you should verify that the memory use is resident and not virtual, which should be very high for large torrents and isn't a problem
deranjer
@deranjer
Jan 25 2018 02:41
Okay, then that. I didn't understand the various storage options so just stuck with all the defaults
Matt Joiner
@anacrolix
Jan 25 2018 02:42
hmmm, try to capture a heap profile then
when it looks big
do you know how to do that?
deranjer
@deranjer
Jan 25 2018 02:43
No, I was going to try an online tool whose name I forget that would do some captures for me and display a bunch of information
but haven't had time to do that yet
https://stackimpact.com/ That was the site
Denis
@elgatito
Jan 25 2018 06:29
@anacrolix , don't forget to commit missinggo/bitmap :) it's not compiling without it :)
Matt Joiner
@anacrolix
Jan 25 2018 06:30
yeah woops :)
it's pushed
Denis
@elgatito
Jan 25 2018 06:33
I was thinking about not-started downloads, where connections arise, but download is not going. can it be caused by bolddb completion which stores completions even when file was deleted?
probably better always to delete completions if user selects to delete the file. but then user can also do that manually and we'll have the same situation, completions stored while file is missing
Matt Joiner
@anacrolix
Jan 25 2018 06:36
yeah that thing you reported should be fixed
setting file priorities with no readers
i don't have tests for a lot of this stuff, i need to expand them
Denis
@elgatito
Jan 25 2018 06:37
with the very last version download started fine, will make few tests
Matt Joiner
@anacrolix
Jan 25 2018 06:37
unfortunate that setting up torrent tests is insanely complicated
cheers
Denis
@elgatito
Jan 25 2018 06:37
can't check unpn, so poor network here :(
Matt Joiner
@anacrolix
Jan 25 2018 06:37
my router is a piece of crap, so i don't think its upnp works properly
:D
Denis
@elgatito
Jan 25 2018 06:40
anyway it should not break downloading, but can help with uploading :)
will wait for reports
Matt Joiner
@anacrolix
Jan 25 2018 06:40
yeah pretty much
i just added the benchmarks to the CI
Denis
@elgatito
Jan 25 2018 06:41
latest release downloaded 1.3k times, so you will have enough reports :D
Matt Joiner
@anacrolix
Jan 25 2018 06:41
damn nice
Denis
@elgatito
Jan 25 2018 07:59
@anacrolix which benchmark can I use for testing boltdb->any_other_db migration to see the difference?
Denis
@elgatito
Jan 25 2018 09:28
@anacrolix , got a report about failed torrent parsing - http://dropmefiles.com/i9Lvv
checked it myself
elgatito@HP:~/Downloads$ torrent-verify test_fail.torrent 
2018/01/25 11:26:55 main.go:75: bencode: value (string) is not appropriate for type: int64
Matt Joiner
@anacrolix
Jan 25 2018 09:42
looks like the creation date is a string
i don't have any benchmarks for storage. it would be super easy to add some since it's an entirely standalone package
that error message isn't very helpful is it
Denis
@elgatito
Jan 25 2018 09:44
about marshal erro?
Matt Joiner
@anacrolix
Jan 25 2018 09:44
yeah
i didn't write the bencode stuff, altho i maintain it now inside the repo
it's been pretty good
Denis
@elgatito
Jan 25 2018 09:59
i see the way to add some parameter like CreationDate int64bencode:"creation date,omitempty,omiterror"`
Matt Joiner
@anacrolix
Jan 25 2018 09:59
that's a good idea
Denis
@elgatito
Jan 25 2018 10:00
or make the field as string and see if it's int then use it, if not - convert
Matt Joiner
@anacrolix
Jan 25 2018 10:00
i might do both
i'll add omiterror, and also create a CreationDate type that handles this peculiarity
Denis
@elgatito
Jan 25 2018 10:01
awesome
Matt Joiner
@anacrolix
Jan 25 2018 10:09
$ godo github.com/anacrolix/torrent/cmd/torrent-verify test_fail.torrent
2018/01/25 21:09:16 main.go:75: parsing value for key "creation date": bencode: value (string) is not appropriate for type: int64
much nicer :)
Denis
@elgatito
Jan 25 2018 10:10
at least if there any more issues with torrent files - we'll know where and why
Matt Joiner
@anacrolix
Jan 25 2018 10:40
yeah
always showing outcoming speed as 0
that is the same torrent info dump
Denis
@elgatito
Jan 25 2018 19:18
we use Torrent.Stats().DataBytesWritten to get bytes count that was sent, and that is always 0
actually have never seen it incrementing
Matt Joiner
@anacrolix
Jan 25 2018 20:30
I should expose the stats in the status too
Denis
@elgatito
Jan 25 2018 21:01
we are counting download rate by reading Torrent.BytesCompleted(), so this value is rising slowly, while bandwidth is used totally
on my linux hosts I don't have such problems
can it be pieces/chunks are downloaded and downloaded again and that is why BytesCompleted() is not changing so fast?