travisbrown/iteratee

Iteratees for cats

Travis Brown

@travisbrown

@sullivan- taking a look now. at a glance I'd say all bets are off if you have mutable state in your enumerator, but the duplication still looks odd.

John Sullivan

@sullivan-

i don't think that embedded iterator is the problem...

my fix is in Done.bind method, line 106 Step.scala

as you can see, f(value) is computed twice. if i extract local, the problem goes away

John Sullivan

@sullivan-

I now have a test that shows it out; unfortunately the test stack overflows for twitter:test/test .. looking into it

  "foreach" should "perform an operation on all values in a grouped stream" in forAll { (eav: EnumeratorAndValues[Int]) =>
     var total = 0
     val iteratee = foreach[Vector[Int]](is => total += is.sum)

     val eavg = EnumeratorAndValues(
       eav.enumerator.grouped(3),
       eav.values.grouped(3).toVector)

     assert(eavg.resultWithLeftovers(iteratee) === F.pure(((), Vector.empty)) && total === eavg.values.flatten.sum)
   }

(in IterateeSuite.scala)

John Sullivan

@sullivan-

weird, i am unable to get twitter:test/test to stack overflow a second time

John Sullivan

@sullivan-

I put in a PR, maybe a better spot to continue conversation travisbrown/iteratee#180

Travis Brown

@travisbrown

@sullivan- got sidetracked just now, but thanks much for the PR—I'll try to get it reviewed this afternoon.

@sullivan- btw I'm considering a new release soon-ish—there are a few little things I've added locally for a personal project that I'll try to push this week, and there are new Monix and Twitter Util releases.

@sullivan- so if there's anything you'd like to see added, now's the time :)

John Sullivan

@sullivan-

I wish I had more for you Travis :smile: reactive streams stuff turned out to be a bit more involved than I expected .. and I'm otherwise sidetracked.
Thanks for you consideration re: PR

John Sullivan

@sullivan-

@travisbrown I still have a lot to learn. The PR I gave you yesterday not stack safe. Unfortunately it only shows up sporadically in the tests - normally for JS. I just updated the PR with a stack safe version

just saw your comment on the PR, will take a look

Travis Brown

@travisbrown

@sullivan- this stuff can be so confusing. thanks for the PR and the fix.

John Sullivan

@sullivan-

Very happy to contribute. It's just about the best way for me to learn this stuff. Thanks for the great project.

John Sullivan

@sullivan-

@travisbrown any chance of a iteratee.io release that includes that PR of mine you merged recently? I'd like to do a release of my own some time soon that depends on that change. Thanks!

Travis Brown

@travisbrown

@sullivan- hmm, I really wanted to get that SO sorted first but haven't had time (I'm at NE Scala right now). Is it just the foreach bug? That should be backported anyway, so maybe we could just cut an 0.9.1 today with that and not publish 0.10.0 with a potential stack bug?

John Sullivan

@sullivan-

@travisbrown that would be totally fine with me. but i can also wait a few days, no big deal. enjoy the conference, i wish i was there. thanks!

Travis Brown

@travisbrown

@sullivan- okay, so I've got a fix for the SO bug (which is a real bug in sequenceI that existed before the foreach fix).

I should be able to get 0.10.0 published tonight.

John Sullivan

@sullivan-

awesome thanks Travis!

John Sullivan

@sullivan-

I'm curious to take a look at your fix..

Travis Brown

@travisbrown

@sullivan- I've just opened #185, which I think is almost ready to go—there's just a weird test failure to address first.

Travis Brown

@travisbrown

Okay! 0.10.0 is on its way to Maven Central: https://github.com/travisbrown/iteratee/releases/tag/v0.10.0

Travis Brown

@travisbrown

Would anyone object if I publish releases more frequently / for more minor stuff? e.g. I've got a few little additions since 0.10 that'd be convenient for me to have in a side project, but they're not binary compatible thanks to pre-2.12 traits.

Travis Brown

@travisbrown

I'm thinking of moving iteratee-files to add a cats-effect dependency (and use it for "suspendible" file modules). Any objections?

P. Oscar Boykin

@johnynek

I'd be a little bummed I think. Wouldn't this mean you can't use Try or Future?

To me, it seems like MonadError plus a few lazy params could do the trick

(Even ApplicativeError in many cases)

By lazy I mean call-by-name or use of Eval.

Travis Brown

@travisbrown

@johnynek No, what I was thinking was that SuspendableFileModule would remain basically unchanged, but that there would be non-module versions of the reading / writing methods for type constructors with the appropriate cats-effect type class instances.

P. Oscar Boykin

@johnynek

That sounds fine then.

Sergey Kolbasov

@sergeykolbasov

@travisbrown Hi there

I have a question regarding .ensure in Enumerator and Iteratee.
Is it called after last step processed or failed?

Let's say, I have an Enumerator of futures, and I want to close connection on reading failure or when I'm done with stream.

Currently it seems like .ensure is called before the last element is processed (if I add it here):
https://github.com/ImLiar/finch/blob/246fd7ccf80e9da9d8fedd409fbf09ce2d7bda18/iteratee/src/main/scala/io/finch/iteratee/package.scala#L40

I got an exception every time when I write inside of the readerWriter,
https://github.com/ImLiar/finch/blob/246fd7ccf80e9da9d8fedd409fbf09ce2d7bda18/iteratee/src/test/scala/io/finch/iteratee/EnumerateEndpointSpec.scala#L45 saying that writer is already closed

Sergey Kolbasov

@sergeykolbasov

I guess I should use ensureEval instead /_-

Sergey Kolbasov

@sergeykolbasov

Yeah, ensureEval with Eval.later did a trick. Now data is properly discarded.

I wonder if it's a normal behaviour for .ensure to have call-by-value argument with Eval.now under the hood?

Travis Brown

@travisbrown

@ImLiar yeah, I'll take a closer look but I think this should depend on the strict/non-strictness of the context type constructor.

My general approach so far has been to make non-strict contexts the first-class citizens, with things like Option and Try supported by explicitly lazy methods with the Eval suffix.

I think I'd like to stick to this approach (unless everyone else strongly disagrees), but it should be better documented.

Travis Brown

@travisbrown

(the primary motivation here is that I don't want to penalize people who are doing the right thing by using Monix's Task or Rerunnable or (soon) cats.effect.IO with extra overhead)

Sergey Kolbasov

@sergeykolbasov

@travisbrown

Could you give a hint if it's possible to make an accumulator in the middle of a stream that will push accumulated results down to rest of a stream each X seconds?

So I need a batching, essentially

Sergey Kolbasov

@sergeykolbasov

And one more question

I have two Enumerators and append one to another. Both of them have ensureEval. What will be the behaviour in that case? Will the ensure happen after both streams are done:
stream1 -> stream2 -> ensure1 -> ensure2
or it's gonna be like:
stream1 -> ensure1 -> stream2 -> ensure2?

Travis Brown

@travisbrown

@ImLiar the ensure on the first stream should happen as soon as its elements are consumed.

P. Oscar Boykin

@johnynek

@travisbrown I remember you talking about removing FileModule. What is the status there? Seems like a good idea to me now. Cats effect is nearing 1.0, has Bracket, Async and Concurrent and seem like all you need.

Then just keep all the code from SuspendableFileModule. I guess purity is the way to go, so we can give up the Try module.

Travis Brown

@travisbrown

@johnynek yes, this is the plan for the next iteratee.io release. I started a branch built on the cats-effect type classes but was planning to wait for the 1.0. I can go ahead and open a PR against the RC this weekend.

Mansur Ashraf

@MansurAshraf

is there a way to go from Enumeratee[F, A,B]] to Enumeratee[G, A,B]] whereF= cats.io and G=monox.task

Mansur Ashraf

@MansurAshraf

@travisbrown ^

Travis Brown

@travisbrown

@MansurAshraf you should be able to do it via mapI on Step if you have natural transformations in both direction, but it's going to be terribly inefficient and there are almost certainly better ways to solve the problem.

Where communities thrive

People

Repo info

Activity