Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 13 05:20

    47erbot on sbt-1.7.1

    (compare)

  • Jul 13 05:20

    47erbot on main

    Update sbt to 1.7.1 (compare)

  • Jul 13 05:20
    47erbot closed #644
  • Jul 13 05:19

    47erbot on cats-effect-3.3.14

    (compare)

  • Jul 13 05:19

    47erbot on main

    Update cats-effect to 3.3.14 (compare)

  • Jul 13 05:19
    47erbot closed #645
  • Jul 13 05:14
    47erbot opened #645
  • Jul 13 05:14

    47erbot on cats-effect-3.3.14

    Update cats-effect to 3.3.14 (compare)

  • Jul 13 05:14
    47erbot opened #644
  • Jul 13 05:14

    47erbot on sbt-1.7.1

    Update sbt to 1.7.1 (compare)

  • Jul 11 05:23

    47erbot on main

    Update sbt to 1.7.0 (compare)

  • Jul 11 05:23
    47erbot closed #643
  • Jul 11 05:23

    47erbot on sbt-1.7.0

    (compare)

  • Jul 11 05:17
    47erbot opened #643
  • Jul 11 05:17

    47erbot on sbt-1.7.0

    Update sbt to 1.7.0 (compare)

  • Jun 29 05:25
    47erbot closed #642
  • Jun 29 05:25

    47erbot on cats-effect-3.3.13

    (compare)

  • Jun 29 05:25

    47erbot on main

    Update cats-effect to 3.3.13 (compare)

  • Jun 29 05:20
    47erbot opened #642
  • Jun 29 05:20

    47erbot on cats-effect-3.3.13

    Update cats-effect to 3.3.13 (compare)

Raúl Raja Martínez
@raulraja
@FloWi thanks for the kind words :clap:
Basti
@lunaryorn
Hi, I'm considering fetch to simplify some of my data fetches accross our services, but I'm facing a question: We've got an ubiquituous "request context" which contains authentication, tracing IDs, etc. When I call a service in a data source I must pass the incoming request context; but I see no "context" support in fetch. How'd I model this?
Raúl Raja Martínez
@raulraja
@lunaryorn hi! how are you currently passing that context in code non related to Fetch? Implicits, Kleisli?. If you can provide a minimal sample code we can probably help you define a DataSource instance that would help you with that context.
Basti
@lunaryorn
@raulraja As an explicit argument; I call someService.someEndpoint(requestContext, someParameter). The context itself comes from the thrift handler, and gets passed by the invoking service.
I can't move to implicits or arrows w/o signifcant refactoring, unfortunately
Justin Heyes-Jones
@justinhj

I'm curious about this too. It seems like you can't extend DataSource to do this since you'd need to change the fetch method itself. @lunaryorn's request seems similar to the way the DataSourceCache works... given a DataSourceIdentity it knows how to check a cache (that is mutated across subsequent fetch executions.

Maybe to solve this kind of issue you'd need a similar implicit type (something like DataSourceContext), which given a DataSourceIdentity knows how to both retrieve or create the context for that entity and also modify it after the fetch.

If that makes sense.

Raúl Raja Martínez
@raulraja
sorry it took me so long to get back to this, I think you can just apply the data source implicitly locally with whatever constructions params you need per call. something along the lines of?
def doInRequestContext(ctx: Context, p: Param): ? = {
   implicit val dataSource = CustomDataSource(ctx, p)
   doWithFetchImplicitDS()
}
wouldn't something like that works to not just in fetch but to inject parameters into any arbitrary type class instances?
Basti
@lunaryorn
@raulraja I thought about this as well; I just wasn't sure whether it's "intended" use of this library, ie, whether we should instantiate data sources locally. But I'll try and see how it goes, many thanks for your help :+1:
Justin Heyes-Jones
@justinhj
Ah, that's a nice and simple solution. I thought the question was how to pass a specific context based on the data source id
Raúl Raja Martínez
@raulraja
I think if type classes where coherent and global in Scala this would be a pain but given you can just use scoped implicits there is no limitation as to the scope in which implicits are contructed and therefore can see their outside world including being provided inside functions that provide the context.
@lunaryorn there is no policies as to intended use so we just follow whatever Scala let's you do for creating data sources. I think if you use global ones you can just provide global implicit instances but in case where you need access to a context like this case they need to be provided in place since global ones would most likely be singletons. You can also provide them globally if you make the ctx and p there be themselves implicits and you only provide those there. This is similar to what play framework does with { implicit request =>
Basti
@lunaryorn
@raulraja I think I'd follow play's approach then, and make data source constructors take implicit parameters for the request context. I'll try and see how it works out for me. Thanks for your help and your recommendations
Raúl Raja Martínez
@raulraja
@lunaryorn sounds good, let us know if you find any shortcomings with that approach, cheers!
Dermot Haughey
@hderms
Will fetch ever have destructive updates or is this always a read oriented system
Raúl Raja Martínez
@raulraja
@hderms at the moment it's read oriented but there have been discussion in the past around allowing writes. Nobody has been motivated enough to implement a POC around that though since does not seem to be a common use case for current users. If you'd like to take a shot at it we could offer guidance with any questions around Fetch.
Binh Nguyen
@ngbinh

If I have one single data source (think of a document db that stores JSON documents). Then the data source can be like:

implict object JsonStore extends DataSource[String, JSON]{
  def name: String = ???
  def fetchOne(id: Identity): Query[Option[Result]] = ???
  def fetchMany(ids: NonEmptyList[Identity]): Query[Map[Identity, Result]] = ???
}

Now, we store more than one kinds of document there. Says, User and Post. We also have UserId and PostId that basically wrap a String and enforce certain format for each ids. And UserModel and PostModel are case classes that could be serialized to JSON. What is the right way to have two data sources that return both data types and can batch and cache results for both?

Thanks

Taleb Zeghmi
@talebzeghmi

Hi,
Playing with introduction code http://47deg.github.io/fetch/docs.html#introduction-0

  for {
    x: User <- getUser(1)
  } yield s"${x.id} ${x.username}

Produces following error:
Error:(105, 23) value filter is not a member of fetch.Fetch[User]

Thoughts on how to get around this?

Taleb Zeghmi
@talebzeghmi

got help in cats room:
@emilypi

because for comprehensions are implemented poorly.
every time you have a predicate on your type like : Type or if foo, it calls withFilter

Emily Pillmore
@emilypi
I was pinged in here :P
@tewf what kind of effect are you dealing with that doens’t have withFilter?
Dermot Haughey
@hderms
@raulraja nah I agree I think in retrospect destructive updates don't make sense with it
Patrick Curran
@patrickthebold_twitter
I just started looking at fetch, and I might have a hard time formulating this question.
I like the deduplication feature, but when I run a fetch I'd like to be able to get the data as it becomes available. (Imagine I have a websocket and I want to push data through it as it becomes available.) My issue is that if I use:
(fetchA,fetchB).tupled even if the two fetches ultimately execute in parallel, I have to wait until they both complete to get the full tuple.
I think I want to be able to run the fetch and get an F[(F[A],F[B])] or something like that.
Patrick Curran
@patrickthebold_twitter
So somehow I'd like to get separate ConcurrentEffects that still share the same underlying batching/caching logic.
Alejandro Gómez
@purrgrammer
hey Patrick, your use case is interesting but not something the library supports at the moment
one of the requirements of the library is that data should fit in memory, so it is fine for constructing HTTP responses, but not a good fit for streaming through websockets or server-sent events
Patrick Curran
@patrickthebold_twitter
I'm not sure this changes anything, but in my case memory is not an issue. I could wait until I have everything and then send to my UI, but I'd like to send some data from the first round of calls, render what I can, and then send subsequent data.
Alejandro Gómez
@purrgrammer

I could wait until I have everything and then send to my UI, but I'd like to send some data from the first round of calls, render what I can, and then send subsequent data.

the use case makes sense, although I'm not sure Fetch could support that at the moment. I've been talking with a colleague about how we could support it but I need to spend some more cycles on it to have an answer

i used Fetch for reading data for a UI, although it was pre-rendered in the server
it then runs the Fetch in the client too in case something has been updated
Patrick Curran
@patrickthebold_twitter

thanks, I just wanted to be clear it wasn't a memory/infinite stream thing I was asking for. Thinking out loud: How about attaching an IO action/callback to a Fetch. Something like:

def subFlatMap[A, B](fa: Fetch[F, A])(f: A => F[B]): Fetch[F, B] = Unfetch(fa.run.flatMap {
      case Done(v) => f(v).map(Done(_))
      case Throw(e) =>
        Applicative[F].pure(Throw[F, B](e))
      case Blocked(br, cont) =>
        Applicative[F].pure(Blocked(br, subFlatMap(cont)(f)))
    })

Then you could do:
```
def sendToClientA: F[A] = for {
_ <- sendData(a)
} yield a
def sendDataA: F[Unit] = ???

Anyway I'll keep thinking about it...(And trying to figure out cats-effect) thanks again!
Ryan Tomczik
@Tomczik76
I have a question about how the Cache works. Does it have a time to live?
Alejandro Gómez
@purrgrammer
@Tomczik76 glad you asked. The default implementation of the cache uses an in-memory cache with no TTL, so all the data in a fetch is kept in the cache. Note that using a cache with a TTL could cause different values to be returned for the same identity. Since we want to keep the guarantee that once you read an identity it will be the same for the whole Fetch execution, we use no TTL in the cache.
Alejandro Gómez
@purrgrammer
see https://github.com/47deg/fetch/blob/master/examples/src/test/scala/JedisExample.scala#L156-L187 for an example of a Redis cache implementation that you can use when running a Fetch. Cache is pluggable so you can write your own with a custom TTL
Bijan Chokoufe Nejad
@bijancn
Hi all. It seems to me that when composing together many Fetches, I can only flatMap or traverse, i.e. there is only support for the happy path, so all Fetches succeed or one fails and then all fail. However, I have in some cases appropriate business logic what to do if it's not there or would even return the user an empty result. The only way I see of doing this is to do bring it back to the IO context and handle the error there with something like Fetch.run(myFetch).attempt. But then I would like to get back to the Fetch[IO, _] monad so it plays together with all the other requests that have to be composed together on the higher level
Bijan Chokoufe Nejad
@bijancn
after looking at the sources, I guess I could do this but now I will have to carry around everywhere a ConcurrentEffect, ContextShift and Timer
  private def fetchOrNone(f: Fetch[IO, ExternalId])(
      implicit C: cats.effect.ConcurrentEffect[IO],
      CS: cats.effect.ContextShift[IO],
      T: cats.effect.Timer[IO]
  ): Fetch[IO, Option[ExternalId]] =
    Unfetch[IO, Option[ExternalId]](
      Fetch
        .run(f)
        .map(Some(_))
        .handleError(_ => None[ExternalId])
        .map(Done(_))
    )
Omer Zach
@omerzach
Hey! Loving what i've seen so far of fetch and experimenting with using it at my company. a bit stuck on one specific question and one more general one and hoping to get a bit of help if someone has a sec.
  1. I'm going through the docs (http://47deg.github.io/fetch/docs.html) in a scala REPL but getting issues calling .tupled on a pair of Fetch[F, Foo] instances:
scala> def fetchProduct[F[_] : ConcurrentEffect]: Fetch[F, (User, User)] =
     |   (getUser(1), getUser(2)).tupled
<console>:200: error: value tupled is not a member of (fetch.Fetch[F,User], fetch.Fetch[F,User])
         (getUser(1), getUser(2)).tupled
                                  ^
am i just missing some imports?
and now more generally, is it possible to use Fetch with a fixed concurrent effect type? our codebase uses cats.effect.IO throughout rather than more generic cats effect typeclasses. can i just fix type FetchIO[T] = Fetch[IO, T] (not sure if that's actually proper syntax for higher kinded programming in scala) and use FetchIO instead of Fetch throughout? we may generalize down the line, but looking for a quicker win first.
Alejandro Gómez
@purrgrammer
hi @omerzach, regarding the use of .tupled, make sure you import the Applicative syntax cats.syntax.applicative._ so you can use it, see https://github.com/typelevel/cats/blob/master/docs/src/main/tut/typeclasses/applicative.md#syntax
regarding whether it's possible to use Fetch with a fixed IO type, you can just do type FetchIO[A] = Fetch[IO, A] as you mentioned :+1:
hey @bijancn, regarding the "happy path", one asumption that Fetch does is that your identities will be there, and will short-circuit when one is missing. However, for optional identities, you can construct Fetch instances using Fetch#optional. It will yield a Fetch[F, Option[A]], and won't fail if the identity is missing. Let me know if it helps!
Alejandro Gómez
@purrgrammer
@omerzach another option would be using Fetch parameterised to F[_] : ConcurrentEffect, and use ConcurrentEffect[F].liftIO in the data sources. this way you can use your IO-returning functions in the data source implementation, and keep your fetch code generic.
Alejandro Gómez
@purrgrammer
just cut the 1.0.0 release of Fetch, for the curious: https://github.com/47deg/fetch/releases/tag/v1.0.0
Pepe García
@pepegar
:clap: :clap: :clap:
Omer Zach
@omerzach

@purrgrammer

scala> import cats.syntax.applicative._
import cats.syntax.applicative._

scala> def fetchProduct[F[_] : ConcurrentEffect]: Fetch[F, (User, User)] =
     |   (getUser(1), getUser(2)).tupled
<console>:181: error: value tupled is not a member of (fetch.Fetch[F,User], fetch.Fetch[F,User])
         (getUser(1), getUser(2)).tupled
                                  ^

after copy-pasting everything in http://47deg.github.io/fetch/docs.html#syntax-6-companion-object-0 up to the def getUser