These are chat archives for vu3rdd/functorrent

4th
Aug 2015
Ramakrishnan Muthukrishnan
@vu3rdd
Aug 04 2015 05:16
wondering if monoid can be used in some way to create a full file from individual pieces. Problem is that Monoids do not seem to have any ordering restrictions.
where as we do, for writing pieces into a file.
Ramakrishnan Muthukrishnan
@vu3rdd
Aug 04 2015 06:56
I guess it is stupid to call it a monoid because the associativity property no longer holds.
Jaseem Abid
@jaseemabid
Aug 04 2015 07:07
I have to check
mappend does have order defined, right?
And do you need associativity to stitch files in order?
Divam
@dfordivam
Aug 04 2015 08:17
@jaseemabid I dont think we can simply let go of synchronous exceptions.
The peer threads share a common resource (Disk IO handle) which has to be a part of ControlThread as an MVar, so peer threads may take a while to finish doing disk IO and therefore ControlThread need to wait for them to exit gracefully
I used killThread to stop the listening thread because there we don't have resource sharing.
How do you plan to use STM, I can't think of a use case where it would be better than MVar
Jaseem Abid
@jaseemabid
Aug 04 2015 15:11
@dfordivam Yes. We cant get rid of sync exceptions completely. Agreed.
No concrete cases where STM is better than MVar for now. but knowing the tool will help us pick it when needed.
Jaseem Abid
@jaseemabid
Aug 04 2015 15:17

Here is the problem: We might kill the thread half way during a write, corrupting the file.

Previous solution: Wait for threads to complete themselves. The thread goes back into main loop after it finishes writing. It gets a new command and if that is STOP, it goes down

Issue with prev solution: pkill will corrupt the file. So will a bad exception or something we cant anticipate today

Current solution: Make threads kill -safe. Let them be killed at any time and let it manage cleanup internally.

Only remaining question, how to make writeFile work nicely with async exceptions?
I'll go figure that out.

@dfordivam Am I clear? Is this what you mean?
One thing I can think of now, is to do the same thing we did for logging for writing files too. Just like we have a logging thread, can we have a disk writer too?
Jaseem Abid
@jaseemabid
Aug 04 2015 15:22
Is writeFile interruptible ? The docs are not very clear
Jaseem Abid
@jaseemabid
Aug 04 2015 16:43
@vu3rdd @dfordivam Any idea about ^ ?
Jaseem Abid
@jaseemabid
Aug 04 2015 16:59
writeFile itself is defined with a bracket.
-- | Write a 'ByteString' to a file.
writeFile :: FilePath -> ByteString -> IO ()
writeFile f txt = bracket (openBinaryFile f WriteMode) hClose
    (\h -> hPut h txt)
Which imply openBinaryFile is interruptible
openBinaryFile :: FilePath -> IOMode -> IO Handle
openBinaryFile fp m =
  catchException
    (openFile' fp m True True)
    (\e -> ioError (addFilePathToIOError "openBinaryFile" fp e))
=> openFile' is also interruptible
openFile' is some low level machinery. I think we can conclude that its indeed interruptible.
Jaseem Abid
@jaseemabid
Aug 04 2015 17:10
@dfordivam @vu3rdd Just curious. Is it viable for us to catch up once in a while? I'm willing to set aside about 30m everyday anytime so that we can talk synchronously
Jaseem Abid
@jaseemabid
Aug 04 2015 17:30

I cant think of anything other than using a writer thread and the same mechanism we used for logging. Also, replace both logging and file writer MVar with a Chanel (a mvar link list) so that read and write are decoupled and a slow disk wont slow down peer threads.

Channels over single mvar is explained well here http://chimera.labs.oreilly.com/books/1230000000929/ch07.html#sec_channels