These are chat archives for anacrolix/torrent

5th
Jan 2018
Matt Joiner
@anacrolix
Jan 05 2018 03:27
Denis' advice is spot on. What are the priorities for in your use case deranjer?
deranjer
@deranjer
Jan 05 2018 03:30
@anacrolix Essentially just so that a user can specify which files they want downloaded first, perhaps it is a massive torrent and they want the first files downloaded first so they can start watching/using them. Or they want to cancel certain files they already have so that they aren't downloading things they don't need.
Matt Joiner
@anacrolix
Jan 05 2018 03:31
yeah ok. i guess most clients offer high/medium/low priorities
deranjer
@deranjer
Jan 05 2018 03:31
yeah I use it a decent amount in uTorrent server for linux which is what this will be replacing since that is no longer maintained
and the web clients out there for other solutions are very poor so decided to try and write my own
Matt Joiner
@anacrolix
Jan 05 2018 03:32
got a link to it?
I don't think the last commit was even in a working state yet
Matt Joiner
@anacrolix
Jan 05 2018 03:32
i meant utorrent server :P
deranjer
@deranjer
Jan 05 2018 03:32
oh lol
Matt Joiner
@anacrolix
Jan 05 2018 03:33
deranjer
@deranjer
Jan 05 2018 03:33
yeah that is it
Matt Joiner
@anacrolix
Jan 05 2018 03:40
I think having more fine-grained download priorities is worthy of an issue
it's also very easy to implement, the code is already designed to handle that kind of thing with readers
deranjer
@deranjer
Jan 05 2018 03:41
Oh okay, do you want me to open it? I'm learning about Readers still (I've been learning go for about 5 months) so I can try and do a PR but can't say the code would be very good.
Matt Joiner
@anacrolix
Jan 05 2018 03:42
no just an issue is fine, so i can collect any specifics people might want
deranjer
@deranjer
Jan 05 2018 03:42
Sounds good I'll open it
Matt Joiner
@anacrolix
Jan 05 2018 03:52
@elgatito is your file_reopen branch about conserving file descriptor use?
Matt Joiner
@anacrolix
Jan 05 2018 04:50
also has that commit resolved most/all of the issues with pieces stalling?
Denis
@elgatito
Jan 05 2018 05:23
@anacrolix , yes, opening file once, then reusing it
about pieces - looks good for some people, but sometimes readers stop sending data to kodi, even I see there are readers and pieces are completed. Would be great if I can reproduce it, but I can't, so leaving it now
still very strange thing on some systems when reader tries to read piece and in few milliseconds client is calling MarkComplete. Looks like sometimes MarkComplete is called after piece is marked as completed internally?
Denis
@elgatito
Jan 05 2018 05:32
also was thinking about priorities. let's say we have a torrent with many file, we want one, and we also want a srt file, so then we set File.Download() and set a reader to that file, but what will happen if readehead for that reader is 50mb and file size is 15kb? Will it download 50mb of the next file in torrent?
Matt Joiner
@anacrolix
Jan 05 2018 06:52
still very strange thing on some systems when reader tries to read piece and in few milliseconds client is calling MarkComplete. Looks like sometimes MarkComplete is called after piece is marked as completed internally?
it doesn't look like that in the code

also was thinking about priorities. let's say we have a torrent with many file, we want one, and we also want a srt file, so then we set File.Download() and set a reader to that file, but what will happen if readehead for that reader is 50mb and file size is 15kb? Will it download 50mb of the next file in torrent?

i believe it will download the next 50mb, for so long as the reader is open anyway. if you read what you wanted and immediately closed, the readahead would be minimized

generally the assumption is if you download via the Download methods, you intend to use the storage yourself directly
or you want to give the client advance notice that you will make use of an entire file or range of pieces
Denis
@elgatito
Jan 05 2018 06:57
It would be perfect to have something like File.Queued bool which with false is allowing only to get first and last piece (to keep verification happy)
anyway, current implementation is fine for me :)
Matt Joiner
@anacrolix
Jan 05 2018 07:00
hm i don't quite follow, can you expand on that?
Denis
@elgatito
Jan 05 2018 07:02
I mean basically we can add simple bool to mark file to be passed by the Client or not, but then if first piece of this file belongs to previous file partially, then we should download that piece
to be able to read it fully
Denis
@elgatito
Jan 05 2018 07:08
the first thought about doing this: add a File.Queued bool, add File.Queue() and File.Unqueue(), which are appending Torrent.QueueuedPieces bitmap, which is checked upon piecespriority update loop
should be fast enough and pretty straight
Matt Joiner
@anacrolix
Jan 05 2018 07:36
i wonder if this can be abstracted on top of the client
Denis
@elgatito
Jan 05 2018 07:39
what you mean? leave File unchanged and possibly do everything in Torrent?
Matt Joiner
@anacrolix
Jan 05 2018 07:40
no no, just thinking at a programming methodology level, often there are things core to an implementation, for example pieces in the case of torrent, that a lot of other abstractions can be built on top, rather than intrusively alongside the rest of the implementation
for example handling files, is just deciding to give ranges of pieces special treatment for convenience
at an implementation level, there could be a minimal core that just handles pieces, then a higher level package that wraps that dealing with readers, files, configuration etc.
not to say this is specific to this implementation, just a general observation. i haven't seen this done in Go much
Matt Joiner
@anacrolix
Jan 05 2018 07:57
okay i just understood your queued thing now, you want a way to avoid reader readahead traversing files?
Denis
@elgatito
Jan 05 2018 08:03
yupp
Matt Joiner
@anacrolix
Jan 05 2018 08:03
if we have a flag on readers, this becomes a super simple change
Denis
@elgatito
Jan 05 2018 08:04
Kodi is requesting data piece-by-piece, and when it comes to the last piece, reader will download all readahead
flag on readers? to say only use current file?
Matt Joiner
@anacrolix
Jan 05 2018 08:04
yep
Reader.LimitReadaheadToFile(bool) or some better name if u can think of one
Denis
@elgatito
Jan 05 2018 08:05
didn't think about it, would work, yes :)
Matt Joiner
@anacrolix
Jan 05 2018 08:05
it would default to true
so you don't get readahead leaking through everytime you create a new reader
or maybe i just add File.NewReader
that has it defaulting to true, otherwise you get false with Torrent.NewReader
that seems better
i think it's time to add File.NewReader anyway, people always get confused about how to do that themselves
Denis
@elgatito
Jan 05 2018 08:06
true, true
i'd also rename Priritize* method as they don't do what naming stands for, but rather just enable download
Matt Joiner
@anacrolix
Jan 05 2018 08:08
shit yeah i didn't notice that
Denis
@elgatito
Jan 05 2018 08:08
maybe a //DEPRECATED comment and a copy-paste to keep compilation fine
Matt Joiner
@anacrolix
Jan 05 2018 08:11
yep good idea
Denis
@elgatito
Jan 05 2018 08:15
many useful things come along with golang :)
Denis
@elgatito
Jan 05 2018 17:12
https://img.devrant.com/devrant/rant/r_177018_3ma1s.jpg - spent a day speeding up caching and move from files to boltdb, with testing of serialization libraries and ended up with MsgPack+ModifiedGzip, moved from 500ms to 1ms... pfff... enough for a tech interview :D
deranjer
@deranjer
Jan 05 2018 20:46
@elgatito I love boltDB, recently started using storm for it since my front end expects Json so easy to store in the databases as Json as well. Storm does support other serialization methods though I think