Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 20 21:40
    codecov-commenter commented #633
  • Nov 20 21:36
    codecov-commenter commented #630
  • Nov 20 21:36
    codecov-commenter commented #628
  • Nov 20 21:36
    scala-steward closed #616
  • Nov 20 21:36
    scala-steward commented #616
  • Nov 20 21:36
    scala-steward opened #633
  • Nov 20 21:36
    codecov-commenter commented #629
  • Nov 20 21:35
    codecov-commenter commented #628
  • Nov 20 21:34
    scala-steward closed #614
  • Nov 20 21:34
    scala-steward commented #614
  • Nov 20 21:34
    scala-steward opened #632
  • Nov 20 21:33
    scala-steward closed #627
  • Nov 20 21:33
    scala-steward commented #627
  • Nov 20 21:33
    scala-steward opened #631
  • Nov 20 21:33
    scala-steward closed #583
  • Nov 20 21:33
    scala-steward commented #583
  • Nov 20 21:33
    scala-steward opened #630
  • Nov 20 21:33
    scala-steward closed #619
  • Nov 20 21:33
    scala-steward commented #619
  • Nov 20 21:33
    scala-steward opened #629
Erik Osheim
@non
ah ok, i see.
Travis Brown
@travisbrown
@non also it got caught by the stack-safety laws as soon as I updated to Cats 0.8.0.
Erik Osheim
@non
well, that's something at least! :)
Travis Brown
@travisbrown
okay, here's my latest shot at the Step change I've been trying to get done for a couple of weeks: #146
@johnynek this is pretty similar to the change you reviewed in #129, but with a couple of additional API changes. if you have any concerns, let me know—I'd like to get this published today so I can also publish the new circe release.
P. Oscar Boykin
@johnynek
I'll take a look.
Travis Brown
@travisbrown
thanks.
P. Oscar Boykin
@johnynek
@travisbrown so, how much do you want me to share allocation anxiety?
For instance, in few places you could do some ugly thing: get an Iterator and then transform as building rather than a few .map/traverse calls.
Travis Brown
@travisbrown
@johnynek the more the better, I guess, but if you're basically okay with the API I might want to push some improvements like that to after 0.8.0.
P. Oscar Boykin
@johnynek
Okay. I'll focus on the non-internal details.
Travis Brown
@travisbrown
@johnynek Any thoughts on #146?
Travis Brown
@travisbrown
iteratee.io 0.7.1 is out with Scala 2.12.0 artifacts.
RomanIakovlev
@RomanIakovlev
Yay!
RomanIakovlev
@RomanIakovlev
Just for the better understanding: results of methods like io.iteratee.files.SuspendableFileModule#readBytesFromStream are not supposed to be consumed multiple times, right? I.e. if I create an Enumerator instance and consume it in a certain way (e.g. taking head of it), it doesn’t make sense to keep that instance, because I’ll not be able to consume it again.
RomanIakovlev
@RomanIakovlev
If the above is correct, I need another piece of advice. I have a big zipped json file (hundreds of MB zipped), which contains an array of json objects. In the tests I want to take an arbitrary entry from that file and do something with it. I want to get the length of the array first, and generate a random number from 0 to length to get the entry for test. However it seems I’ll have to read through the file twice: once to get the length, then to get the entry. Since the file is huge, it takes time, and tests are running slower. Maybe there’s a way to do this in one go?
Travis Brown
@travisbrown
@RomanIakovlev All enumerators provided by the library should be side-effect free, so you can re-use them however you need.
@RomanIakovlev for that application specifically, it sounds like reservoir sampling would do what you're looking for (it's not provided by iteratee, but it's a pretty straightforward algorithm).
RomanIakovlev
@RomanIakovlev
Okay, I need to take a deeper look. I’ve seen some stream already closed exceptions when tried to reuse the enumerators, but it might be just my sloppy coding skills.
Travis Brown
@travisbrown
hmm, if you can reproduce I'd definitely be interested in making sure it's not a logic problem.
RomanIakovlev
@RomanIakovlev
sure
will try to reproduce
RomanIakovlev
@RomanIakovlev
Yeah, and thanks @travisbrown for mentioning the reservoir sampling. I didn’t know about such term.
Travis Brown
@travisbrown
yesterday on the plane I threw this together: #153
it allows us to work with strings as chunks without creating Seq[Char]s all over the place.
any feedback appreciated, but the motivation will be clearer once I open the iteratee-parsing PR (should be today or tomorrow).
Travis Brown
@travisbrown
I'm planning to publish 0.8.0 today, with some small additions (#157), version updates for Monix and Twitter Util, and 2.12 support for iteratee-twitter (but not #153). If anyone would like to see any other changes in this version, please let me know asap.
Adelbert Chang
@adelbertc
Was the Task type removed from Iteratee?
I could've sworn the library provided a Task type
are you just using Eval for effect capture? suspicious eyes
@travisbrown
Adelbert Chang
@adelbertc
also im trying to repeatedly run a () => Future[A] to get an infinite Enumerator[Future, A] - is this provided somewhere, or is this consciously not provided for Future
Travis Brown
@travisbrown
@adelbertc for iteratee-core it's not really (much) of an issue—you just bring your own type (the more task-like the better).
in iteratee-files we have an interim solution where you have modules that play a role similar to Capture and Async type classes. You create an instance of a module and get file I/O enumerators, etc., and part of creating the module instance is that you have to handle effect capture.
@adelbertc you're probably looking for Enumerator.liftMEval. I was skeptical about providing first-class support in core for effect capture for types like Future, but Oscar talked me into a couple of concessions, and liftMEval is one of them.
Adelbert Chang
@adelbertc
ah yes, perhaps that
i shall try that
how do i "run" an Enumerator
Adelbert Chang
@adelbertc
oh looks like its toVector
hm Enumerator.liftMEval(Eval.always { Future { println("hi"); 5 }}).take(5).toVector doesnt seemt o do what i think it does
in that it only evaluates once
Travis Brown
@travisbrown
@adelbertc once you've got a future you're back in the horrible world of future semantics, but if you stick with Enumerator[Future, A] you're okay.
Adelbert Chang
@adelbertc
so what you're saying is i should use scalaz.concurrent.Task if i want something sane
Travis Brown
@travisbrown
well, iteratee.io does its best—it just can't help you once you run the iteratee if what you're running it in is Future.
@adelbertc e.g.
scala> import cats.Eval, cats.instances.future._
import cats.Eval
import cats.instances.future._

scala> import io.iteratee.Enumerator
import io.iteratee.Enumerator

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> val enum = Enumerator.liftMEval(Eval.always { Future { println("hi"); 5 }}).take(5)
enum: io.iteratee.Enumerator[scala.concurrent.Future,Int] = io.iteratee.Enumeratee$$anon$25@ddca1ad

scala> enum.toVector
res0: scala.concurrent.Future[Vector[Int]] = List()

scala> hi
enum.toVector
hi
res1: scala.concurrent.Future[Vector[Int]] = List()
(note that enum.toVector is just a convenience method for having the enumerator run an iteratee that collects all values.)
Adelbert Chang
@adelbertc
@travisbrown so the API i have is basically () => Future[List[A]] and i want to keep doing that "forever"
so i have a Stream[A]
er, Enumerator[Future, A]
or Enumerator[Task, A] if thats less terrible