Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 05 2019 14:43
    @typelevel-bot banned @jdegoes
  • Jan 31 2019 21:17
    codecov-io commented #484
  • Jan 31 2019 21:08
    scala-steward opened #484
  • Jan 31 2019 18:19
    andywhite37 commented #189
  • Jan 31 2019 02:41
    kamilongus starred typelevel/cats-effect
  • Jan 30 2019 00:01
    codecov-io commented #483
  • Jan 29 2019 23:51
    deniszjukow opened #483
  • Jan 29 2019 23:37
  • Jan 29 2019 23:22
  • Jan 29 2019 20:26
    Rui-L starred typelevel/cats-effect
  • Jan 29 2019 18:01
    jdegoes commented #480
  • Jan 29 2019 17:04
    thomaav starred typelevel/cats-effect
  • Jan 28 2019 17:43
    asachdeva starred typelevel/cats-effect
  • Jan 28 2019 07:12
    alexandru commented #480
  • Jan 28 2019 05:45
    codecov-io commented #482
  • Jan 28 2019 05:35
    daron666 opened #482
  • Jan 27 2019 13:56
    codecov-io commented #481
  • Jan 27 2019 13:46
    lrodero opened #481
  • Jan 27 2019 05:47
    codecov-io commented #460
  • Jan 27 2019 05:37
    codecov-io commented #460
Adam Rosien
@arosien

So. Question. My understanding is that the hypothetical ideal for an IOApp is to have more or less the exact number of threads in the execution context as available CPUs. However, this doesn't necessarily take into account whatever threads back things like blaze or asynchronous channel groups and whatnot, even completely ignoring blocking operations. Is there any guidance for correct configuration of pool sizing in the face of some of these other non-blocking pools.

@rzeigler https://typelevel.org/cats-effect/concurrency/basics.html#choosing-thread-pool has the basics, along with the daniel's gist (linked in the cats-effect page also).

generally, you shouldn't have to tune the default contextshift, if you only use it for cpu-bound operations. anything that blocks should be shifted to a Blocker.

doobie (for database access) provides a good, concrete example. it requires separate pools for cpu, (blocking) JDBC io, and different one for connection management. https://tpolecat.github.io/doobie/docs/14-Managing-Connections.html#about-threading

Adam Rosien
@arosien

something to read: at first i was like, "eww!", because kubernetes, but this is a good set of slides about using cats-effect

https://speakerdeck.com/alexeyn/writing-kubernetes-operators-using-cats-effect

Anton Sviridov
@keynmol
@arosien in another example, cats-effect was really useful for deflaking AWS API in several different situations - lots of random failures from the API breaking the state and the flow if done via the official API. We used it to manage resource scaling in multiple settings, it was quite natural to model retries, consistency and maintaining a linear flow you can actually read :)
Adam Rosien
@arosien
i'd love to learn more!
Anton Sviridov
@keynmol
I'd love to share more, with actual substance :D But in general some things in AWS land - like Kinesis scaling, ECS provisioning, Athena query execution, etc. - they're all ripe for being handled by, really, any effect monad, where you're updating in multiple AWS regions, where the steps are the same, but each operation is a minefield of dozens of potential errors from AWS SDK. Recoverable, retriable, and not

Kinesis Scaling is a particularly interesting one - it takes time, you need to do it in multiples of two, you want the AWS state to "settle" before you move on, and AWS doesn't support multi-region operations, so you want to parallelize as much as possible

Using cats-effect there was the least amount of magic - the whole thing fits on a single screen and there's no black box, like terraform, which does something that achieves similar things

Adam Rosien
@arosien
sounds super. and that last part would be a great quote :)
i guess i mean i'm interested in examples where particular types or combinators modeled those behaviors. i'm guessing using Async to map whatever concurrency the library used into IO, etc.
Anton Sviridov
@keynmol

Yeah, to work with SDK Async and ContextShift are important - to wrap CompletableFeature from AWS and also to ensure that one never accidentally shifts to AWS' internal threadpool and stays there to execute the rest of tasks

There' Concurrent to have Ref and Deferred which are important to to maintain state updated in parallel (specifically when you want multiple regions to reach a particular state before you move on to the next stage)

Adam Rosien
@arosien
ah, the "accidentally did work on the library's thread pool" problem, a classic
Daniel Spiewak
@djspiewak
my eternal shame…
Anton Sviridov
@keynmol

I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.

Some sort of internal testing that guarantees that program flow always returns to the most performant task pool would go a long way.

For an internal presentation I demonstrated that via tracking the execution pool in a concurrent map, but I wonder if current work on IO and Fiber tracing can make this better

Bob Glamm
@glammr1

to wrap CompletableFeature from AWS and also to ensure that one never accidentally shifts to AWS' internal threadpool

Huh, I encountered that annoying behavior in the AWS Java SDK about six months ago when consuming SQS messages

Anton Sviridov
@keynmol
@glammr1 I think using Bracket's guarantee will help - if you have a ContextShift in scope, you can shift back to it regardless of how the AWS action completed
Bob Glamm
@glammr1
I'll take a look. I don't remember if I ended up solving that or if it ended up not making a difference in my use case
Daniel Spiewak
@djspiewak

I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.

Observing and testing it is a huge pain. It can be done relatively easily in the types though: https://github.com/typelevel/cats-effect/blob/ce3/core/src/main/scala/cats/effect/Async.scala#L26-L32 That's not a hard and fast guarantee, but at least the error-prone bit is on the implementor's shoulders, not on the user's.

softshipper
@softshipper

Could someone please explain me:

def apply[F[_]](implicit ev: Environment[F]): ev.type = ev

what .type is?

Daniel Spiewak
@djspiewak
It's tricky Scala type computation stuff :-)
Basically it means "the precise type of ev, whatever that happens to be"
It's easier to see with a smaller example
softshipper
@softshipper
for example:
trait Environment[F[_]] {
  def get(v: String) : F[Option[String]]
}

object Environment {

  def apply[F[_]](implicit ev: Environment[F]): ev.type = ev

  def impl[F[_]: Sync]: Environment[F] = new Environment[F] {
    override def get(v: String): F[Option[String]] =
      Sync[F].delay(sys.env.get(v))
  }
}
Daniel Spiewak
@djspiewak
trait Foo
class Bar extends Foo

def test1(f: Foo): Foo = f
def test2(f: Foo): f.type = f

val b1: Bar = new Bar

val b2 = test1(b1)    // => : Foo
val b3 = test2(b1)    // => : Bar
softshipper
@softshipper
the apply function with ev.type what does it mean?
Daniel Spiewak
@djspiewak
it means "return the instance of Environment discovered by implicit search, but give it its specifically declared type, rather than the type Environment"
softshipper
@softshipper
@djspiewak thanks a lot
Daniel Spiewak
@djspiewak
np
pool
@hamstakilla
is there any way i can lift SomeInstance[F] into SomeInstance[IO]?
Christopher Davenport
@ChristopherDavenport
If someInstance[F] has mapK or FunctorK and your F has a LiftIO
Sorry, not liftIO, Effect which lets you compile an F to an IO
pool
@hamstakilla

Its basically a

trait SomeInstance[F[_]]

Which was created with F[_]: Sync: ContextShift: Async

Christopher Davenport
@ChristopherDavenport
You need some way to allow SomeInstance[F] => F ~> G => SomeInstance[G]
You can often write this yourself with unsealed traits and if F always apears in covariant position.
pool
@hamstakilla
I should look at kleisi right?
Christopher Davenport
@ChristopherDavenport
Not generally
trait Foo[F[_]]{
    def foo: F[Int]
}
object Foo{
   def mapK[F[_], G[_]](base: Foo[F], fk: F ~> G): Foo[G] =new MapKFoo(base, fk)
    private class MapKFoo[F[_], G[_]](base: Foo[F], fk: F ~> G) extends Foo[G] {
        def foo: G[Int] = fk(base.foo)
    }
}
pool
@hamstakilla
oh i see
thank you
Sean Kwak
@cosmir17
Screenshot 2020-06-20 at 18.46.54.png
Hi, I am doing some scala migration work for a non-scala project (weekend hobby). I don't know what to do with the red circled part. I guess I am to use Ref? but even that I don't think it's the right design.
Does Cats effect dev ever create a class including a shared data store?
pool
@hamstakilla
why not Ref tho?
Dmitry Polienko
@nigredo-tori

@cosmir17, I'd replace a mutable Map with a Ref containing the current state as an immutable Map. But as you said, the whole thing does look messy. Idiomatic FP Scala code tends to emphasize operations over pure data types (case classes and the like) in favor of OOP-style encapsulation of mutable data. So you'd have something like

final case class Board(rowCount: Int, columnCount: Int, grid: Map[Point, GoString[_ <: Player]])
// Some error type you can properly report
final case class IllegalMove(...)
def placeStone[Color <: Player](point: Point, board: Board): Either[InvalidMove, (Board, NeighbourInfo[Color])]
// Same thing, but might be easier to use.
def placeStone[Color <: Player](point: Point): StateT[Either[InvalidMove, *], Board, NeighbourInfo[Color]]

I've gotten rid of parameterization by F here, since it doesn't seem to be necessary.
By the way, parameterizing stuff with Player subtypes looks contrived. It doesn't seem to do anything yet, and it makes expressing certain things difficult. For example, adjacentOppositeColor should actually be a Set[GoString[<opposite of Color>]], but we can't conveniently express that.

Ben Spencer
@dangerousben
@cosmir17 just to double check - does this map actually need to be accessed concurrently? the fact that it was originally just a HashMap kind of implies not?
Sean Kwak
@cosmir17
Thank you @nigredo-tori. I put F there for two reasons.
  1. placeStone operation seems complicated(multiple fold and etc) and will take a long time (around 300ms?).
  2. I am doing it as a practice for Cats effect
Ben Spencer
@dangerousben
if it does though, you could the creation of and access to a ConcurrentHashMap or similar in your Sync effect
Sean Kwak
@cosmir17
@dangerousben It was a python project.
Ben Spencer
@dangerousben
that's quite a lot of code but it looks like it broadly does what I'm talking about
Sean Kwak
@cosmir17
@hamstakilla does Ref natively work for HashMap (or ConcurrentHashMap)?
Can anyone explain why 'https://gist.github.com/johnynek/a8b1a70ebbfaac2836068f8ba3087a72#file-refmap-scala' was written when we can just use Ref? Is it for extra APIs?