Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 05 2019 14:43
    @typelevel-bot banned @jdegoes
  • Jan 31 2019 21:17
    codecov-io commented #484
  • Jan 31 2019 21:08
    scala-steward opened #484
  • Jan 31 2019 18:19
    andywhite37 commented #189
  • Jan 31 2019 02:41
    kamilongus starred typelevel/cats-effect
  • Jan 30 2019 00:01
    codecov-io commented #483
  • Jan 29 2019 23:51
    deniszjukow opened #483
  • Jan 29 2019 23:37
  • Jan 29 2019 23:22
  • Jan 29 2019 20:26
    Rui-L starred typelevel/cats-effect
  • Jan 29 2019 18:01
    jdegoes commented #480
  • Jan 29 2019 17:04
    thomaav starred typelevel/cats-effect
  • Jan 28 2019 17:43
    asachdeva starred typelevel/cats-effect
  • Jan 28 2019 07:12
    alexandru commented #480
  • Jan 28 2019 05:45
    codecov-io commented #482
  • Jan 28 2019 05:35
    daron666 opened #482
  • Jan 27 2019 13:56
    codecov-io commented #481
  • Jan 27 2019 13:46
    lrodero opened #481
  • Jan 27 2019 05:47
    codecov-io commented #460
  • Jan 27 2019 05:37
    codecov-io commented #460
paul-snively-exa
@paul-snively-exa
(Cue @djspiewak: "'Unbounded' is just another word for 'DOS attack.'")
Rafał Sumisławski
@RafalSumislawski
I expect the threads of blocker to spend most of the time blocked, not on the cores, but I see what you mean. The risk of the pool growing infinitly is a problem indeed.
paul-snively-exa
@paul-snively-exa
Exactly.
At some point, when developing services at scale, I think you get radicalized about managing scarce resources explicitly.
Rafał Sumisławski
@RafalSumislawski
Then maybe the recommended practice should be to use thread pools of limited size under the Blocker? I feel like the API and the docs kind of encourage the unlimited cachedThreadPool approach.
paul-snively-exa
@paul-snively-exa
I think so too, but that's why, I think, the "don't use it implicitly" warning is there.
That is, I'd add the warning to ContextShift rather than removing it from Blocker.
Rafał Sumisławski
@RafalSumislawski
I get it.
paul-snively-exa
@paul-snively-exa
tl;dr threads are a scarce resource.
But meh. I take your point, too, about expecting most of them to be blocked rather than actually consuming a core. That's not indefensible, any more than expecting not to OOM, acting as if memory were unbounded, is.
It's all trade-offs.
Fabio Labella
@SystemFw
I think you can apply the "implicit call site" strategy to this problem
(re: implicit Blocker)
same that applies to stateful tagless algebras
basically the idea is that passing things implicitly does not really cause issues, it's implicit instances that can be troublesome (because you can pick up the wrong one by accident)
but if you pass something implicitly, but only make the correspondent instance implicit close to the call site, then it's ok
drawbacks:
  • the language isn't great for this syntax wise, so you have some unwieldy bits, which are normally limited to main however
  • it's not a clear cut rule, so if your team disagrees, just passing things explicitly works better
Fabio Labella
@SystemFw

@danielkarch hey,

Aren't Pure/FlatMap + the run loop essentially a trampoline?

well, you need to introduce an async boundary (or more), and then the execution context essentially acts as a trampoline (which is why Future is stack safe, it submits all the time). However having a fully async boundary (i.e. an ExecutionContext submission) is heavyweight, you can build a lighter weight one (this is trampolineEc in cats-effect).
The thing with stack-safety is that there are some corner cases (you will see comments in the cats-effect codebase like "we need to insert another async boundary here for stack-safety")

Rafał Sumisławski
@RafalSumislawski
My team ins currently on the implicit side. Before there was Blocker in cats-effect we used to create our own wrappers around diffrent ExecutionContext, to be able to pass them around implicitly without the risk of using the wrong implicit instance.
Fabio Labella
@SystemFw
you (@danielkarch) can look at various tests for stack-safety in cats-effect for these various corners
also, you will notice that in my code there is a comment saying "no double execution check" on Async, so my async there is not idempotent (which it has to be)
all that being said, I'm planning to expand on that IO for blog post series and/or a playground for the new cancelation semantics (unfortunately the optimisations in cats-effect proper make the code quite convoluted and hard to experiment with, at least for me)
Rafał Sumisławski
@RafalSumislawski
@djspiewak @paul-snively @SystemFw thanks sharing your opinions on the topic of implicit Blockers :+1:
Daniel Karch
@danielkarch
@SystemFw thank you, I will look at the tests.
Nik Gushchin
@Dr-Nikson

Hello guys!)
I’ve started exploring world of FP recently, and now I’m trying tu build my first http-service with http4s.
And I get these debug messages in my console (on every request that consul does):

Request headers: Headers(Host: docker.for.mac.localhost:7000, User-Agent: Consul Health Check, Accept: text/plain, text/*, */*, Accept-Encoding: gzip, Connection: close)
[21:31:24.850] [scala-execution-context-global-40               ] DEBUG - closeConnection()
[21:31:24.850] [scala-execution-context-global-40               ] DEBUG - Shutting down HttpPipeline
[21:31:24.850] [scala-execution-context-global-40               ] DEBUG - Canceled request
[21:31:24.850] [scala-execution-context-global-40               ] DEBUG - Shutting down.
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Shutting down HttpPipeline
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Canceled request
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Shutting down.
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Shutting down idle timeout stage
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Shutting down.
[21:31:24.851] [scala-execution-context-global-40               ] DEBUG - Shutting down.
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Stage NIO1HeadStage sending inbound command: Disconnected
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Shutting down idle timeout stage
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Shutting down.
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Stage IdleTimeoutStage sending inbound command: Disconnected
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Shutting down HttpPipeline
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Canceled request
[21:31:24.851] [blaze-selector-7                                ] DEBUG - Shutting down.

My code looks like:

 def apply[F[_]: Sync: Async](port: Int)(
    implicit cs: ConcurrentEffect[F],
    timer: Timer[F],
  ): F[Http4sPrometheusExporter] = {

    for {
      prometheusService <- Http4sPrometheusService.build[F]
      httpApp = Router[F]("/" -> prometheusService.routes).orNotFound
      server <- BlazeServerBuilder[F]
        .bindHttp(port, "0.0.0.0")
        .withHttpApp(httpApp)
        .resource
        .use(_ => Async[F].never[Unit])
        .start
    } yield new Http4sPrometheusExporter()
  }

Is it ok? Or I’m duing something wrong?
Appreciate any help :)

BTW api-endpoint seems to be working properly
Rafał Sumisławski
@RafalSumislawski
@Dr-Nikson https://gitter.im/http4s/http4s would be a better place to ask this question.
Nik Gushchin
@Dr-Nikson
@RafalSumislawski ok, thanks)
vimalaguti
@vimalaguti
Hi, what if I want to learn how to use the ConcurrentEffect type class? I'm not sure how it relates with IO.
Gavin Bisesi
@Daenyth
@vimalaguti tldr version: You need Effect when you want to call .unsafeToFuture() or .unsafeRunSync() and Concurrent when you need concurrent execution, racing, timeouts, etc. ConcurrentEffect is both together
vimalaguti
@vimalaguti
Thanks, but somewhat cryptic. Maybe I just need to refresh the difference between type classes and data types, any clue?
Gavin Bisesi
@Daenyth
A typeclass describes a set of behaviors and laws that establish how those behaviors relate to each other
for example the Functor typeclass describes the behavior of map: F[A] => (A => B) => F[B] and laws like composition; fa.map(f).map(g) == fa.map(f andThen g)
vimalaguti
@vimalaguti
but the same is valid for IO, isn't it?
Gavin Bisesi
@Daenyth
"the same" as in?
Ryan Peters
@sloshy
IO extends implements Functor (indirectly, it's a distant parent in the hierarchy)
Oleg Pyzhcov
@oleg-py
It doesn't. IO has an instance of a Functor, and it's not done via inheritance.
Ryan Peters
@sloshy
Err - right, poor choice of words on my part (was thinking of the typeclass hierarchy and got a little mixed up when the words came out)
Oleg Pyzhcov
@oleg-py
There are things you can't have with inheritance nicely like Monoid.zero and Applicative.pure
Oleg Pyzhcov
@oleg-py

@vimalaguti you want to use ConcurrentEffect, like any other typeclasses, when you don't want to be tied to a concrete IO in your signatures.
Specifically, Effect is used when you need to interop with some library that calls YOUR code, such as APIs that let you register a callback (almost everything in Scala.JS) and Concurrent is used where you want flow control with fibers and .start, or for data structures with asynchronous blocking, like Semaphore, Deferred, fs2's Queue, monix ConcurrentChannel, etc.

ConcurrentEffect bundles the two to avoid ambiguity issues (they both extend Async which would break extension methods), and also gives you a run, but give me a cancel token on top of Effect.

IO has some intricacies - you can't do io.start without implicit ContextShift[IO] in scope, but you can always unsafeRunAsync it. For instance, Monix Task requires a Scheduler for unsafeRunAsync but you can always do task.start. ConcurrentEffect lets you write code that is independent of these details and works for all effect types in the same way.

It isn't, however, a typeclass you want to use often, it literally says "here be dragons, anything can happen", and also you can't use it with anything less powerful than IO+ContextShift, or a Task+Scheduler. Something that only requires Sync, for example, can be used with SyncIO or OptionT[IO, ?]

But if your goal isn't to write effect-agnostic code, it's possible to just use IO directly. And if it is, ConcurrentEffect is a typeclass of last resort.
vimalaguti
@vimalaguti
thanks so much to everyone for the explanation. I just need to wrap a concurrent effect (a memory copy) and suspend it until a global slot gets available - eg not used by other memory-copies. As far as I can understand, the non blocking class is the Async, so that I can run many of them, and use Deferred as data type?
vimalaguti
@vimalaguti
Like I have 2 slots, and I want to precharge the next once the slot is free again. But than I need to Await - ah too many overlapping type classes
Oleg Pyzhcov
@oleg-py
Deferred sounds like a good approach, and if you want to be able to concurrently cancel the acquisition somehow, you'd need Concurrent.cancelable builder
The Async doesn't let you "run many of them" out of the box - if your underlying API is thread-blocking, you can't get around with thread management, but you should be able to only use Sync[F] + ContextShift[F] + Blocker
You might need to have Concurrent on the outside, to manage two slots independently, but that is irrelevant
Oleg Pyzhcov
@oleg-py
Async[F] only means "I can lift anything that is calling a callback once, that also doesn't care much about cancelability, into F".
So, it depends on the API you're wrapping - and if you're making your own, Concurrent is probably better since you get cancelable version of Deferred when you have it
vimalaguti
@vimalaguti
Thanks!
I'll try with Concurrent and Deferred - although I don't mind about cancelability, just concurrency.