I figured I better bring this topic to iteratee channel. As mentioned in Circe channel, I want to produce json with Circe in a streaming fashion. I’ve tried to start and faced one question. Basically I need a function like this (imports omitted):
def writeJson[T: Encoder](enum: Enumerator[Task, T], file: File): Task[Unit] = {
val printer = Printer.noSpaces.copy(dropNullKeys = true)
val opener = Enumerator.enumOne[Task, String]("[")
val closer = Enumerator.enumOne[Task, String]("]")
val entries = enum.map(_.asJson.pretty(printer) + ",")
opener.append(entries).append(closer).into(writeLines(file))
}The problem is the comma after the last entry, which makes the resulting json invalid. Is there a way to somehow introspect the Enumerator and to know if that’s the last entry, to handle it differently?
Iteratee, Enumeratee and Enumerator, what they do on their own and how they interact with each other. IIRC you’ve promised a blog post about Iteratee architecture some time ago (no pressure! :smile: ) , but in absence of it, could you please explain here what they are and what they do, in general?
scala> import cats.Monad
import cats.Monad
scala> import io.iteratee.{ Enumeratee, Enumerator, Iteratee }
import io.iteratee.{Enumeratee, Enumerator, Iteratee}
scala> def sortBy[F[_]: Monad, A, B: Ordering](f: A => B): Enumeratee[F, A, A] =
| Enumeratee.sequenceI(Iteratee.consume[F, A]).map(_.sortBy(f)).andThen(Enumeratee.flatMap(Enumerator.enumVector[F, A]))
sortBy: [F[_], A, B](f: A => B)(implicit evidence$1: cats.Monad[F], implicit evidence$2: Ordering[B])io.iteratee.Enumeratee[F,A,A]
scala> import cats.instances.option._
import cats.instances.option._
scala> import io.iteratee.modules.option._
import io.iteratee.modules.option._
scala> enumVector(Vector("a", "aaa", "aa")).through(sortBy((_: String).length)).toVector
res0: Option[Vector[String]] = Some(Vector(a, aa, aaa))
def toEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F]): Enumerator[F, E] = {
new Enumerator[F, E] {
final def apply[A](step: Step[F, E, A]): F[Step[F, E, A]] = {
if (iterator.hasNext) {
F.flatMap(step.feedEl(iterator.next))(s => apply[A](s))
} else {
F.pure(step)
}
}
}
}
@sullivan- that's part of it, but it's more about the mutability breaking referential transparency even when you're working just with the enumerator itself—e.g. even if you're reading a file like this:
val e = io.iteratee.monix.task.readLines(new java.io.File("build.sbt"))you can reuse e as many times as you like and never have to worry about the internal state of the enumerator, etc.
toEnumerator.
Got it! Thanks, that makes sense. So if I changed the signature from
def toCatsEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F])to
def toCatsEnumerator[F[_], E](iteratorGen: () => Iterator[E])(implicit F: Monad[F])i could theoretically get around that problem by calling iteratorGen in the right place inside the Enumerator, right?
