you could wrap all of that work up in an enumeratee, I guess—I haven't yet just because it's more straightforward than the decoding side.
@RomanIakovlev I could imagine some navigation methods on the decoding enumeratees that allow you to navigate into the structure before streaming starts…
But it forces the rigid structure of array of objects though. It’s tolerable for now, because I own both producer and consumer side.
@travisbrown I’m going to give an internal tech talk for my team at work about my experience using Circe streaming with Iteratee. I’m trying to find a good and concise definitions of Iteratee’s main classes:
Iteratee, Enumeratee and Enumerator, what they do on their own and how they interact with each other. IIRC you’ve promised a blog post about Iteratee architecture some time ago (no pressure! :smile: ) , but in absence of it, could you please explain here what they are and what they do, in general?
@RomanIakovlev depending on the audience enumerator = stream, enumerate = transformer, iteratee = fold (or sink) might work.
s/enumerate/enumeratee
@teodimoff this isn't too terrible, though:
scala> import cats.Monad
import cats.Monad
scala> import io.iteratee.{ Enumeratee, Enumerator, Iteratee }
import io.iteratee.{Enumeratee, Enumerator, Iteratee}
scala> def sortBy[F[_]: Monad, A, B: Ordering](f: A => B): Enumeratee[F, A, A] =
| Enumeratee.sequenceI(Iteratee.consume[F, A]).map(_.sortBy(f)).andThen(Enumeratee.flatMap(Enumerator.enumVector[F, A]))
sortBy: [F[_], A, B](f: A => B)(implicit evidence$1: cats.Monad[F], implicit evidence$2: Ordering[B])io.iteratee.Enumeratee[F,A,A]
scala> import cats.instances.option._
import cats.instances.option._
scala> import io.iteratee.modules.option._
import io.iteratee.modules.option._
scala> enumVector(Vector("a", "aaa", "aa")).through(sortBy((_: String).length)).toVector
res0: Option[Vector[String]] = Some(Vector(a, aa, aaa))
circe coming in 3… 2… 1…
def toEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F]): Enumerator[F, E] = {
new Enumerator[F, E] {
final def apply[A](step: Step[F, E, A]): F[Step[F, E, A]] = {
if (iterator.hasNext) {
F.flatMap(step.feedEl(iterator.next))(s => apply[A](s))
} else {
F.pure(step)
}
}
}
}
@sullivan- that's part of it, but it's more about the mutability breaking referential transparency even when you're working just with the enumerator itself—e.g. even if you're reading a file like this:
val e = io.iteratee.monix.task.readLines(new java.io.File("build.sbt"))you can reuse e as many times as you like and never have to worry about the internal state of the enumerator, etc.
@sullivan- that wouldn't be the case for an enumerator you'd get from
toEnumerator.
Got it! Thanks, that makes sense. So if I changed the signature from
def toCatsEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F])to
def toCatsEnumerator[F[_], E](iteratorGen: () => Iterator[E])(implicit F: Monad[F])i could theoretically get around that problem by calling iteratorGen in the right place inside the Enumerator, right?
They are almost entirely equivalent.
The former is syntactically easier to use.
(Of course as far as the types are concerned
Iterable[A] and () => A are equivalent.)
Sorry, not at a computer—meant
() => Iterator[A] two messages up.