def foo = {
Logger[F].info("hello")
otherActionInF
}def foo = {
"irrelevant string"
"actual string that is returned"
}
Logger[F].info("hello") >> otherActionInF
Logger[ConnectionIO] back (I hadn't seen the transact at first)
): F[Int] = {
val contentId = for {
logger <- Slf4jLogger.create[ConnectionIO]
metadata <- contentMeta.toJson.compactPrint.pure[ConnectionIO]
_ <- logger.info(f"Storing content metadata against mid $mid: $metadata")
contentId <- ContentSql.insertMetadata(metadata)
_ <- logger.info("Inserted metadata generating content id $contentId")
_ <- ContentSql.connectMemoryToContent(mid, contentId)
_ <- logger.info(s"Connected content id $contentId to memory mid $mid")
} yield contentId
contentId.transact(xa)
}>> to your existing code without any other changes
info >> {
for {
} yield
}.transact
So. Question. My understanding is that the hypothetical ideal for an IOApp is to have more or less the exact number of threads in the execution context as available CPUs. However, this doesn't necessarily take into account whatever threads back things like blaze or asynchronous channel groups and whatnot, even completely ignoring blocking operations. Is there any guidance for correct configuration of pool sizing in the face of some of these other non-blocking pools.
@rzeigler https://typelevel.org/cats-effect/concurrency/basics.html#choosing-thread-pool has the basics, along with the daniel's gist (linked in the cats-effect page also).
generally, you shouldn't have to tune the default contextshift, if you only use it for cpu-bound operations. anything that blocks should be shifted to a Blocker.
doobie (for database access) provides a good, concrete example. it requires separate pools for cpu, (blocking) JDBC io, and different one for connection management. https://tpolecat.github.io/doobie/docs/14-Managing-Connections.html#about-threading
something to read: at first i was like, "eww!", because kubernetes, but this is a good set of slides about using cats-effect
https://speakerdeck.com/alexeyn/writing-kubernetes-operators-using-cats-effect
Kinesis Scaling is a particularly interesting one - it takes time, you need to do it in multiples of two, you want the AWS state to "settle" before you move on, and AWS doesn't support multi-region operations, so you want to parallelize as much as possible
Using cats-effect there was the least amount of magic - the whole thing fits on a single screen and there's no black box, like terraform, which does something that achieves similar things
Async to map whatever concurrency the library used into IO, etc.
Yeah, to work with SDK Async and ContextShift are important - to wrap CompletableFeature from AWS and also to ensure that one never accidentally shifts to AWS' internal threadpool and stays there to execute the rest of tasks
There' Concurrent to have Ref and Deferred which are important to to maintain state updated in parallel (specifically when you want multiple regions to reach a particular state before you move on to the next stage)
I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.
Some sort of internal testing that guarantees that program flow always returns to the most performant task pool would go a long way.
For an internal presentation I demonstrated that via tracking the execution pool in a concurrent map, but I wonder if current work on IO and Fiber tracing can make this better
I think the main two issues are 1) how hard it is to encode in types 2) how hard it is to observe (and test) on the JVM.
Observing and testing it is a huge pain. It can be done relatively easily in the types though: https://github.com/typelevel/cats-effect/blob/ce3/core/src/main/scala/cats/effect/Async.scala#L26-L32 That's not a hard and fast guarantee, but at least the error-prone bit is on the implementor's shoulders, not on the user's.