Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 05 2019 14:43
    @typelevel-bot banned @jdegoes
  • Jan 31 2019 21:17
    codecov-io commented #484
  • Jan 31 2019 21:08
    scala-steward opened #484
  • Jan 31 2019 18:19
    andywhite37 commented #189
  • Jan 31 2019 02:41
    kamilongus starred typelevel/cats-effect
  • Jan 30 2019 00:01
    codecov-io commented #483
  • Jan 29 2019 23:51
    deniszjukow opened #483
  • Jan 29 2019 23:37
  • Jan 29 2019 23:22
  • Jan 29 2019 20:26
    Rui-L starred typelevel/cats-effect
  • Jan 29 2019 18:01
    jdegoes commented #480
  • Jan 29 2019 17:04
    thomaav starred typelevel/cats-effect
  • Jan 28 2019 17:43
    asachdeva starred typelevel/cats-effect
  • Jan 28 2019 07:12
    alexandru commented #480
  • Jan 28 2019 05:45
    codecov-io commented #482
  • Jan 28 2019 05:35
    daron666 opened #482
  • Jan 27 2019 13:56
    codecov-io commented #481
  • Jan 27 2019 13:46
    lrodero opened #481
  • Jan 27 2019 05:47
    codecov-io commented #460
  • Jan 27 2019 05:37
    codecov-io commented #460
Fabio Labella
@SystemFw
def foo = {
  Logger[F].info("hello")
  otherActionInF
}
which again are both pure values
so it's the same as
def foo = {
  "irrelevant string"
  "actual string that is returned"
}
Gerry Fletcher
@gerryfletch
Ah
Of course
That clears it up a lot
Fabio Labella
@SystemFw
if you want the first "string" to have an influence on the other, you need to explicitly build it
so actually the easiest way is
Logger[F].info("hello") >> otherActionInF
therefore I take my previous statement on Logger[ConnectionIO] back (I hadn't seen the transact at first)
does that make sense?
Gerry Fletcher
@gerryfletch
The first part does, now that I know >> is just syntax for flatmap, so I understand that is how we are combining it into a single value - I'm not sure why you now think ConnectionIO shouldn't be used
As I ended up using it:
  ): F[Int] = {
    val contentId = for {
      logger <- Slf4jLogger.create[ConnectionIO]
      metadata <- contentMeta.toJson.compactPrint.pure[ConnectionIO]
      _ <- logger.info(f"Storing content metadata against mid $mid: $metadata")
      contentId <- ContentSql.insertMetadata(metadata)
      _ <- logger.info("Inserted metadata generating content id $contentId")
      _ <- ContentSql.connectMemoryToContent(mid, contentId)
      _ <- logger.info(s"Connected content id $contentId to memory mid $mid")
    } yield contentId

    contentId.transact(xa)
  }
Fabio Labella
@SystemFw

I'm not sure why you now think ConnectionIO shouldn't be used

ah no, I didn't mean that

I meant that I said "the easiest thing to do is use Logger[ConnectionIO]", but actually you can simply add >> to your existing code without any other changes
Gerry Fletcher
@gerryfletch
Ah, I see
Thank you so much for explaining it all
Fabio Labella
@SystemFw
I hadn't seen transact at first, so I thought you wanted to mix ConnectionIO and F, but you are transacting instead
Gerry Fletcher
@gerryfletch
It makes so much sense now
You have helped me so much, I owe you :laughing:
Fabio Labella
@SystemFw
:)
Gerry Fletcher
@gerryfletch
Is there a shortcut for my above code btw? Storing the result of a for comprehension in a val then calling a method on it? Other than wrapping the comprehension in brackets, or putting it in a separate function
Fabio Labella
@SystemFw
I generally use brackets creatively for that
so
info >> {
   for {
   } yield
}.transact
Gerry Fletcher
@gerryfletch
Very nice
That had not occurred to me
Thanks again :wink:
Fabio Labella
@SystemFw
note that you can say a = foo instead of a <- foo.pure
Gerry Fletcher
@gerryfletch
Omg
Also did not know that
Fabio Labella
@SystemFw
it does not work on the first line of for, which is very annoying
Adam Rosien
@arosien

So. Question. My understanding is that the hypothetical ideal for an IOApp is to have more or less the exact number of threads in the execution context as available CPUs. However, this doesn't necessarily take into account whatever threads back things like blaze or asynchronous channel groups and whatnot, even completely ignoring blocking operations. Is there any guidance for correct configuration of pool sizing in the face of some of these other non-blocking pools.

@rzeigler https://typelevel.org/cats-effect/concurrency/basics.html#choosing-thread-pool has the basics, along with the daniel's gist (linked in the cats-effect page also).

generally, you shouldn't have to tune the default contextshift, if you only use it for cpu-bound operations. anything that blocks should be shifted to a Blocker.

doobie (for database access) provides a good, concrete example. it requires separate pools for cpu, (blocking) JDBC io, and different one for connection management. https://tpolecat.github.io/doobie/docs/14-Managing-Connections.html#about-threading

Adam Rosien
@arosien

something to read: at first i was like, "eww!", because kubernetes, but this is a good set of slides about using cats-effect

https://speakerdeck.com/alexeyn/writing-kubernetes-operators-using-cats-effect

Anton Sviridov
@keynmol
@arosien in another example, cats-effect was really useful for deflaking AWS API in several different situations - lots of random failures from the API breaking the state and the flow if done via the official API. We used it to manage resource scaling in multiple settings, it was quite natural to model retries, consistency and maintaining a linear flow you can actually read :)
Adam Rosien
@arosien
i'd love to learn more!
Anton Sviridov
@keynmol
I'd love to share more, with actual substance :D But in general some things in AWS land - like Kinesis scaling, ECS provisioning, Athena query execution, etc. - they're all ripe for being handled by, really, any effect monad, where you're updating in multiple AWS regions, where the steps are the same, but each operation is a minefield of dozens of potential errors from AWS SDK. Recoverable, retriable, and not

Kinesis Scaling is a particularly interesting one - it takes time, you need to do it in multiples of two, you want the AWS state to "settle" before you move on, and AWS doesn't support multi-region operations, so you want to parallelize as much as possible

Using cats-effect there was the least amount of magic - the whole thing fits on a single screen and there's no black box, like terraform, which does something that achieves similar things

Adam Rosien
@arosien
sounds super. and that last part would be a great quote :)
i guess i mean i'm interested in examples where particular types or combinators modeled those behaviors. i'm guessing using Async to map whatever concurrency the library used into IO, etc.
Anton Sviridov
@keynmol

Yeah, to work with SDK Async and ContextShift are important - to wrap CompletableFeature from AWS and also to ensure that one never accidentally shifts to AWS' internal threadpool and stays there to execute the rest of tasks

There' Concurrent to have Ref and Deferred which are important to to maintain state updated in parallel (specifically when you want multiple regions to reach a particular state before you move on to the next stage)

Adam Rosien
@arosien
ah, the "accidentally did work on the library's thread pool" problem, a classic
Daniel Spiewak
@djspiewak
my eternal shame…
Anton Sviridov
@keynmol

I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.

Some sort of internal testing that guarantees that program flow always returns to the most performant task pool would go a long way.

For an internal presentation I demonstrated that via tracking the execution pool in a concurrent map, but I wonder if current work on IO and Fiber tracing can make this better

Bob Glamm
@glammr1

to wrap CompletableFeature from AWS and also to ensure that one never accidentally shifts to AWS' internal threadpool

Huh, I encountered that annoying behavior in the AWS Java SDK about six months ago when consuming SQS messages

Anton Sviridov
@keynmol
@glammr1 I think using Bracket's guarantee will help - if you have a ContextShift in scope, you can shift back to it regardless of how the AWS action completed
Bob Glamm
@glammr1
I'll take a look. I don't remember if I ended up solving that or if it ended up not making a difference in my use case
Daniel Spiewak
@djspiewak

I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.

Observing and testing it is a huge pain. It can be done relatively easily in the types though: https://github.com/typelevel/cats-effect/blob/ce3/core/src/main/scala/cats/effect/Async.scala#L26-L32 That's not a hard and fast guarantee, but at least the error-prone bit is on the implementor's shoulders, not on the user's.

softshipper
@softshipper

Could someone please explain me:

def apply[F[_]](implicit ev: Environment[F]): ev.type = ev

what .type is?

Daniel Spiewak
@djspiewak
It's tricky Scala type computation stuff :-)
Basically it means "the precise type of ev, whatever that happens to be"