I am thinking of a case where I have an Enumerator on a Free monad and I want to convert that to given monad. But maybe I just need to convert the function used to generate the Enumerator (but if I can do that with a FunctionK it seems like many Enumerators would also have that property.
If I actually look on something other than my phone I might make some headway. ;)
@johnynek you'd think it would be easy to lift an enumerator into a larger context, and I've never been able to come up with a clear intuition about why it's not. Scalaz's
EnumeratorP is one attempt to address this—you define an EnumeratorP[E, F] and then you can apply it to get an EnumeratorT for any larger monad.
@johnynek I asked a question about this a few months ago and got no response: https://twitter.com/travisbrown/status/709108132476293120
Hello everyone. I’m getting strange runtime errors from monix+iteratee+circe combination, and it’s probably the binary incompatibility between those. I have the following now:
val monixVersion = "2.0.4"
libraryDependencies ++= Seq(
"io.monix" %% "monix",
"io.monix" %% "monix-cats"
).map(_ % monixVersion)
val circeVersion = "0.5.3"
libraryDependencies ++= Seq(
"io.circe" %% "circe-core",
"io.circe" %% "circe-generic",
"io.circe" %% "circe-parser",
"io.circe" %% "circe-streaming",
"io.circe" %% "circe-literal"
).map(_ % circeVersion)
val iterateeVersion = "0.6.1"
libraryDependencies ++= Seq(
"io.iteratee" %% "iteratee-files",
"io.iteratee" %% "iteratee-core",
"io.iteratee" %% "iteratee-monix"
).map(_ % iterateeVersion)Are these supposed to work together?
These are the latest
I’ll cross-post to circe, it might be a more active room
https://github.com/travisbrown/iteratee/blob/master/files/src/main/scala/io/iteratee/files/FileModule.scala#L27 - shouldn't
r.close() here be properly captured? Is there really a need to do a NonFatal pattern matching considering that any adequate monad shouldn't catch fatal exception? And is non-exhaustive matching intentional?
catchNonFatal should be used only for side-effect free functions that can throw exceptions (e.g. they might divide by zero but shouldn't do println) AFAIU, since it does pure(a) which is strict in its argument and therefore executes whatever side-effects a: => A might have.
heads up—
tailRecM isn't necessary stack safe when you have thousands of iteratees sequenced on a single chunk if the underlying flatMap isn't stack-safe: #142. it's fixed in #141, but I'm not planning to backport it to an 0.6.2 unless someone is particularly interested, since it's somewhat unlikely people will run into this.
just makes me uneasy that we still need a concept of stack-safe monad independent of
tailRecM
but that's not iteratees' fault.
has more to do with how MTL works
(i.e. i think other people will get bitten by this in other libraries as well.)
@johnynek this is pretty similar to the change you reviewed in #129, but with a couple of additional API changes. if you have any concerns, let me know—I'd like to get this published today so I can also publish the new circe release.
For instance, in few places you could do some ugly thing: get an Iterator and then transform as building rather than a few .map/traverse calls.
Just for the better understanding: results of methods like
io.iteratee.files.SuspendableFileModule#readBytesFromStream are not supposed to be consumed multiple times, right? I.e. if I create an Enumerator instance and consume it in a certain way (e.g. taking head of it), it doesn’t make sense to keep that instance, because I’ll not be able to consume it again.
If the above is correct, I need another piece of advice. I have a big zipped json file (hundreds of MB zipped), which contains an array of json objects. In the tests I want to take an arbitrary entry from that file and do something with it. I want to get the length of the array first, and generate a random number from 0 to length to get the entry for test. However it seems I’ll have to read through the file twice: once to get the length, then to get the entry. Since the file is huge, it takes time, and tests are running slower. Maybe there’s a way to do this in one go?
@RomanIakovlev for that application specifically, it sounds like reservoir sampling would do what you're looking for (it's not provided by iteratee, but it's a pretty straightforward algorithm).
will try to reproduce
it allows us to work with strings as chunks without creating
Seq[Char]s all over the place.
any feedback appreciated, but the motivation will be clearer once I open the iteratee-parsing PR (should be today or tomorrow).
