Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 13 20:21
    scala-steward closed #569
  • Jan 13 20:21
    scala-steward commented #569
  • Jan 13 20:21
    scala-steward opened #588
  • Jan 11 15:12
    scala-steward closed #570
  • Jan 11 15:12
    scala-steward commented #570
  • Jan 11 15:12
    scala-steward opened #587
  • Jan 09 05:55
    codecov-commenter commented #586
  • Jan 09 05:54
    codecov-commenter commented #586
  • Jan 09 05:51
    scala-steward closed #585
  • Jan 09 05:51
    scala-steward commented #585
  • Jan 09 05:51
    scala-steward opened #586
  • Jan 01 04:58
    codecov-commenter commented #585
  • Jan 01 04:57
    codecov-commenter commented #585
  • Jan 01 04:55
    scala-steward closed #584
  • Jan 01 04:55
    scala-steward commented #584
  • Jan 01 04:55
    scala-steward opened #585
  • Dec 31 2021 03:19
    codecov-commenter commented #584
  • Dec 31 2021 03:19
    codecov-commenter commented #584
  • Dec 31 2021 03:16
    scala-steward closed #581
  • Dec 31 2021 03:16
    scala-steward commented #581
RomanIakovlev
@RomanIakovlev
This blog post definitely helps! I guess my questions are pretty much covered there. I had somehow missed it before.
Teodor Dimov
@teodimoff
i tried the library and i liked it ... 10x
Teodor Dimov
@teodimoff
@travisbrown i was looking at fs2 , but i am a fan of your work :D
Travis Brown
@travisbrown
@teodimoff thanks! there are plenty of things fs2 can do that are out of scope for this project, but I find this model works for a lot of the stuff I need, and it's simpler / faster.
Teodor Dimov
@teodimoff
Agreed. The three abstractions are composable enough and flexible enough to make things happen... quickly
Teodor Dimov
@teodimoff
@travisbrown closest thing to Iteratee.sortBy(w => w.length -> w)? i want the hole file sorted by length and by alphabetic order.
Travis Brown
@travisbrown
@teodimoff there's not a super nice way, since all of the off-the-shelf enumeratees don't require gathering all elements in memory…
@teodimoff this isn't too terrible, though:
scala> import cats.Monad
import cats.Monad

scala> import io.iteratee.{ Enumeratee, Enumerator, Iteratee }
import io.iteratee.{Enumeratee, Enumerator, Iteratee}

scala> def sortBy[F[_]: Monad, A, B: Ordering](f: A => B): Enumeratee[F, A, A] =
     |   Enumeratee.sequenceI(Iteratee.consume[F, A]).map(_.sortBy(f)).andThen(Enumeratee.flatMap(Enumerator.enumVector[F, A]))
sortBy: [F[_], A, B](f: A => B)(implicit evidence$1: cats.Monad[F], implicit evidence$2: Ordering[B])io.iteratee.Enumeratee[F,A,A]

scala> import cats.instances.option._
import cats.instances.option._

scala> import io.iteratee.modules.option._
import io.iteratee.modules.option._

scala> enumVector(Vector("a", "aaa", "aa")).through(sortBy((_: String).length)).toVector
res0: Option[Vector[String]] = Some(Vector(a, aa, aaa))
Teodor Dimov
@teodimoff
@travisbrown nope its not bad at all... thx
Travis Brown
@travisbrown
Srepfler Srdan
@schrepfler
:clap:
circe coming in 3… 2… 1…
Travis Brown
@travisbrown
@schrepfler yep :smile:
John Sullivan
@sullivan-
so i wrote this method to convert a scala.collection.Iterator into a io.iteratee.Enumerator. it seems to be working, but i wonder if ppl would be willing to look it over to see what you think? I am still pretty shaky with this cats stuff...
   def toEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F]): Enumerator[F, E] = {
     new Enumerator[F, E] {
       final def apply[A](step: Step[F, E, A]): F[Step[F, E, A]] = {
         if (iterator.hasNext) {
           F.flatMap(step.feedEl(iterator.next))(s => apply[A](s))
         } else {
           F.pure(step)
         }
       }
     }
   }
John Sullivan
@sullivan-
@travisbrown i would especially appreciate your opinion
Travis Brown
@travisbrown
@sullivan- looks reasonable to me (you could add chunking, etc., but that's just an optimization). the only reason we don't provide something exactly like that is because the resulting enumerator inherits the mutability of the iterator.
John Sullivan
@sullivan-
thanks @travisbrown ! when you say "inherits the mutability" do you mean something like other holders of the Iterator might call methods on the iterator, which would cause the enumerator to behave differently?
Travis Brown
@travisbrown

@sullivan- that's part of it, but it's more about the mutability breaking referential transparency even when you're working just with the enumerator itself—e.g. even if you're reading a file like this:

val e = io.iteratee.monix.task.readLines(new java.io.File("build.sbt"))

you can reuse e as many times as you like and never have to worry about the internal state of the enumerator, etc.

@sullivan- that wouldn't be the case for an enumerator you'd get from toEnumerator.
John Sullivan
@sullivan-

Got it! Thanks, that makes sense. So if I changed the signature from

def toCatsEnumerator[F[_], E](iterator: => Iterator[E])(implicit F: Monad[F])

to

def toCatsEnumerator[F[_], E](iteratorGen: () => Iterator[E])(implicit F: Monad[F])

i could theoretically get around that problem by calling iteratorGen in the right place inside the Enumerator, right?

Alexander Konovalov
@alexknvl
That's still not enough since iteratorGen could be returning the same iterator each time.
John Sullivan
@sullivan-
so my assumption is that the caller of this toCatsEnum method knows they have to provide a function that is going to produce a new, equivalent iterator each time. e.g., i could put that in the scaladoc comment
John Sullivan
@sullivan-
@travisbrown you're right, chunking would help. its taking like 20 seconds to process 5M ints..
John Sullivan
@sullivan-
(for the record, that 20 seconds was due to a bad fold somewhere else)
Alexander Konovalov
@alexknvl
@sullivan- Then it doesn't really matter if you take your Iterator by => or by () =>.
They are almost entirely equivalent.
The former is syntactically easier to use.
John Sullivan
@sullivan-
thanks @alexknvl . yeah that was just sloppy on my part
Travis Brown
@travisbrown
@sullivan- @alexknvl I'd probably use Iterable since it basically means "something that can produce an iterator" but has the additional conventional requirement that it's a fresh one.
(Of course as far as the types are concerned Iterable[A] and () => A are equivalent.)
Travis Brown
@travisbrown
Also I think the syntactic ease of use of => Iterator[A] is exactly what I'd not want here, personally.
Sorry, not at a computer—meant () => Iterator[A] two messages up.
John Sullivan
@sullivan-
that totally makes sense, thanks Travis. It turns out what I actually want is something more like () => Iterator[A] with java.io.Closeable, so the enumerator is able to call iter.close() when the enumeration is done early.
John Sullivan
@sullivan-
Hey @travisbrown, I just wanted to let you know I got a version now that produces a reusable Enumerator from an Iterator! (I'm not expecting you to care that much, I'm just excited and have to share :smile: ) https://github.com/longevityframework/unblocking/blob/master/core/src/main/scala/unblocking/ToCatsEnumerator.scala
Travis Brown
@travisbrown
@sullivan- :+1:
John Sullivan
@sullivan-
it occurred to me that converters from/to org.reactivestreams.Publisher[A] would probably be much more sensible than converters from/to () => Iterator[A] with Closeable
Travis Brown
@travisbrown
@sullivan- I'd like to have a reactivestreams conversions module—just haven't had the time to work on one myself (hint, hint :smile:).
John Sullivan
@sullivan-
I'm probably going to write one soon. I'd be happy to contribute it to your codebase. I'd probably need your assistance in figuring out where to put impls and tests, and how to integrate these things with your codebase. How about I write an implementation in my own repo first, and then if you like it, we can talk about how to get it into yours?
There is also this whole TCK Java-API testing framework thingie that I'm not sure how much I want to deal with.. I guess I ought to take a look in the least http://www.reactive-streams.org/reactive-streams-tck-1.0.0-javadoc/
Travis Brown
@travisbrown
@sullivan- :+1:, sounds good to me!
John Sullivan
@sullivan-

@travisbrown would you consider this a bug? I've managed to get Iteratee.foreach to replay some elements twice

https://gist.github.com/sullivan-/4561d9fce97ca0fafdc187d17755bf8f

(If you think it's a bug, I have a simple fix. Alas no test yet.)

Travis Brown
@travisbrown
@sullivan- taking a look now. at a glance I'd say all bets are off if you have mutable state in your enumerator, but the duplication still looks odd.
John Sullivan
@sullivan-
i don't think that embedded iterator is the problem...

my fix is in Done.bind method, line 106 Step.scala

as you can see, f(value) is computed twice. if i extract local, the problem goes away

John Sullivan
@sullivan-
I now have a test that shows it out; unfortunately the test stack overflows for twitter:test/test .. looking into it
  "foreach" should "perform an operation on all values in a grouped stream" in forAll { (eav: EnumeratorAndValues[Int]) =>
     var total = 0
     val iteratee = foreach[Vector[Int]](is => total += is.sum)

     val eavg = EnumeratorAndValues(
       eav.enumerator.grouped(3),
       eav.values.grouped(3).toVector)

     assert(eavg.resultWithLeftovers(iteratee) === F.pure(((), Vector.empty)) && total === eavg.values.flatten.sum)
   }
(in IterateeSuite.scala)
John Sullivan
@sullivan-
weird, i am unable to get twitter:test/test to stack overflow a second time
John Sullivan
@sullivan-
I put in a PR, maybe a better spot to continue conversation travisbrown/iteratee#180