https://github.com/travisbrown/iteratee/blob/master/files/src/main/scala/io/iteratee/files/FileModule.scala#L27 - shouldn't
r.close() here be properly captured? Is there really a need to do a NonFatal pattern matching considering that any adequate monad shouldn't catch fatal exception? And is non-exhaustive matching intentional?
catchNonFatal should be used only for side-effect free functions that can throw exceptions (e.g. they might divide by zero but shouldn't do println) AFAIU, since it does pure(a) which is strict in its argument and therefore executes whatever side-effects a: => A might have.
heads up—
tailRecM isn't necessary stack safe when you have thousands of iteratees sequenced on a single chunk if the underlying flatMap isn't stack-safe: #142. it's fixed in #141, but I'm not planning to backport it to an 0.6.2 unless someone is particularly interested, since it's somewhat unlikely people will run into this.
just makes me uneasy that we still need a concept of stack-safe monad independent of
tailRecM
but that's not iteratees' fault.
has more to do with how MTL works
(i.e. i think other people will get bitten by this in other libraries as well.)
@johnynek this is pretty similar to the change you reviewed in #129, but with a couple of additional API changes. if you have any concerns, let me know—I'd like to get this published today so I can also publish the new circe release.
For instance, in few places you could do some ugly thing: get an Iterator and then transform as building rather than a few .map/traverse calls.
Just for the better understanding: results of methods like
io.iteratee.files.SuspendableFileModule#readBytesFromStream are not supposed to be consumed multiple times, right? I.e. if I create an Enumerator instance and consume it in a certain way (e.g. taking head of it), it doesn’t make sense to keep that instance, because I’ll not be able to consume it again.
If the above is correct, I need another piece of advice. I have a big zipped json file (hundreds of MB zipped), which contains an array of json objects. In the tests I want to take an arbitrary entry from that file and do something with it. I want to get the length of the array first, and generate a random number from 0 to length to get the entry for test. However it seems I’ll have to read through the file twice: once to get the length, then to get the entry. Since the file is huge, it takes time, and tests are running slower. Maybe there’s a way to do this in one go?
@RomanIakovlev for that application specifically, it sounds like reservoir sampling would do what you're looking for (it's not provided by iteratee, but it's a pretty straightforward algorithm).
will try to reproduce
it allows us to work with strings as chunks without creating
Seq[Char]s all over the place.
any feedback appreciated, but the motivation will be clearer once I open the iteratee-parsing PR (should be today or tomorrow).
I could've sworn the library provided a
Task type
are you just using
Eval for effect capture? suspicious eyes
@travisbrown
in iteratee-files we have an interim solution where you have modules that play a role similar to
Capture and Async type classes. You create an instance of a module and get file I/O enumerators, etc., and part of creating the module instance is that you have to handle effect capture.
@adelbertc you're probably looking for
Enumerator.liftMEval. I was skeptical about providing first-class support in core for effect capture for types like Future, but Oscar talked me into a couple of concessions, and liftMEval is one of them.