Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 30 06:13

    47erbot on scalatest-3.2.14

    (compare)

  • Sep 30 06:13

    47erbot on main

    Update scalatest to 3.2.14 (compare)

  • Sep 30 06:13
    47erbot closed #658
  • Sep 30 06:05
    47erbot opened #658
  • Sep 30 06:05

    47erbot on scalatest-3.2.14

    Update scalatest to 3.2.14 (compare)

  • Sep 28 06:11

    47erbot on mdoc-2.3.5

    (compare)

  • Sep 28 06:11

    47erbot on main

    Update mdoc, sbt-mdoc to 2.3.5 (compare)

  • Sep 28 06:11
    47erbot closed #657
  • Sep 28 06:03

    47erbot on mdoc-2.3.5

    Update mdoc, sbt-mdoc to 2.3.5 (compare)

  • Sep 28 06:03
    47erbot opened #657
  • Sep 26 06:14

    47erbot on main

    Update mdoc, sbt-mdoc to 2.3.4 (compare)

  • Sep 26 06:14

    47erbot on mdoc-2.3.4

    (compare)

  • Sep 26 06:14
    47erbot closed #656
  • Sep 26 06:06

    47erbot on mdoc-2.3.4

    Update mdoc, sbt-mdoc to 2.3.4 (compare)

  • Sep 26 06:06
    47erbot opened #656
  • Sep 23 06:01

    47erbot on scala-library-2.13.9

    (compare)

  • Sep 23 06:01

    47erbot on main

    Update scala-library to 2.13.9 (compare)

  • Sep 23 06:01
    47erbot closed #655
  • Sep 23 05:55

    47erbot on scala-library-2.13.9

    Update scala-library to 2.13.9 (compare)

  • Sep 23 05:55
    47erbot opened #655
Aarsh Shah
@aarshkshah1992
Also, any list of planned features / issues/ bug fixes I can work on to get my foot in the door ? Would really appreciate the help.
Raúl Raja Martínez
@raulraja
no guide at the moment since it's a self contained and small library, feel free to glance over the issues and pick up any of the current open ones and ask any questions here :)
Aarsh Shah
@aarshkshah1992
Great, thanks a lot :smile:
Raúl Raja Martínez
@raulraja
:thumbsup:
Tom Adams
@tomjadams
Silly question, has anyone tried to use fetch from java? Or, know a similar library that’s usable from java?
kerr
@hepin1989
@tomjadams You will need to wrote a wrapper around it
Tom Adams
@tomjadams
@hepin1989 Thanks, that’s what I figured.
Raúl Raja Martínez
@raulraja
Java has no way to represent scala higher kinds and some of the methods would need to be made concrete or specialized. Same for the implicit resolution would have to be wrapped in a scala project that exposes concrete types and already resolved instances with signatures compatible in Java. If you plan on going down that rabbit hole we'd welcome such a submodule in Fetch itself
You probably want ListenableFuture<A> as result type from guava, we have instances to go to that and back in Freestyle async so a Fetch in terms of scala.concurrent.Future can just be ~> to guava's ListenableFuturepotentially.
Tom Adams
@tomjadams
Thanks for the info @raulraja. The async story in Java is pretty ordinary.
We have a lot of code that mixes concurrency with data access, and some of that data access has hand rolled caches. We also have some future need to be able to see the input state of some data into (the equivalent of) a Fetch, and I think the patterns in Fetch would suit this quite well.
We’re also looking at other things like atomix, I don’t know enough yet though about our use case and future use case to be able to make concrete plans though.
Raúl Raja Martínez
@raulraja
Cool!, as mentioned if you end up building this atop Fetch let us know and we'll help in whatever we can. 😀
Tom Adams
@tomjadams
👍🏼
Florian Witteler
@FloWi
Hey guys! Thanks for creating fetch! Made my day :) Also kudos to that awesome documentation. Can't wait to give it a try
Raúl Raja Martínez
@raulraja
@FloWi thanks for the kind words :clap:
Basti
@lunaryorn
Hi, I'm considering fetch to simplify some of my data fetches accross our services, but I'm facing a question: We've got an ubiquituous "request context" which contains authentication, tracing IDs, etc. When I call a service in a data source I must pass the incoming request context; but I see no "context" support in fetch. How'd I model this?
Raúl Raja Martínez
@raulraja
@lunaryorn hi! how are you currently passing that context in code non related to Fetch? Implicits, Kleisli?. If you can provide a minimal sample code we can probably help you define a DataSource instance that would help you with that context.
Basti
@lunaryorn
@raulraja As an explicit argument; I call someService.someEndpoint(requestContext, someParameter). The context itself comes from the thrift handler, and gets passed by the invoking service.
I can't move to implicits or arrows w/o signifcant refactoring, unfortunately
Justin Heyes-Jones
@justinhj

I'm curious about this too. It seems like you can't extend DataSource to do this since you'd need to change the fetch method itself. @lunaryorn's request seems similar to the way the DataSourceCache works... given a DataSourceIdentity it knows how to check a cache (that is mutated across subsequent fetch executions.

Maybe to solve this kind of issue you'd need a similar implicit type (something like DataSourceContext), which given a DataSourceIdentity knows how to both retrieve or create the context for that entity and also modify it after the fetch.

If that makes sense.

Raúl Raja Martínez
@raulraja
sorry it took me so long to get back to this, I think you can just apply the data source implicitly locally with whatever constructions params you need per call. something along the lines of?
def doInRequestContext(ctx: Context, p: Param): ? = {
   implicit val dataSource = CustomDataSource(ctx, p)
   doWithFetchImplicitDS()
}
wouldn't something like that works to not just in fetch but to inject parameters into any arbitrary type class instances?
Basti
@lunaryorn
@raulraja I thought about this as well; I just wasn't sure whether it's "intended" use of this library, ie, whether we should instantiate data sources locally. But I'll try and see how it goes, many thanks for your help :+1:
Justin Heyes-Jones
@justinhj
Ah, that's a nice and simple solution. I thought the question was how to pass a specific context based on the data source id
Raúl Raja Martínez
@raulraja
I think if type classes where coherent and global in Scala this would be a pain but given you can just use scoped implicits there is no limitation as to the scope in which implicits are contructed and therefore can see their outside world including being provided inside functions that provide the context.
@lunaryorn there is no policies as to intended use so we just follow whatever Scala let's you do for creating data sources. I think if you use global ones you can just provide global implicit instances but in case where you need access to a context like this case they need to be provided in place since global ones would most likely be singletons. You can also provide them globally if you make the ctx and p there be themselves implicits and you only provide those there. This is similar to what play framework does with { implicit request =>
Basti
@lunaryorn
@raulraja I think I'd follow play's approach then, and make data source constructors take implicit parameters for the request context. I'll try and see how it works out for me. Thanks for your help and your recommendations
Raúl Raja Martínez
@raulraja
@lunaryorn sounds good, let us know if you find any shortcomings with that approach, cheers!
Dermot Haughey
@hderms
Will fetch ever have destructive updates or is this always a read oriented system
Raúl Raja Martínez
@raulraja
@hderms at the moment it's read oriented but there have been discussion in the past around allowing writes. Nobody has been motivated enough to implement a POC around that though since does not seem to be a common use case for current users. If you'd like to take a shot at it we could offer guidance with any questions around Fetch.
Binh Nguyen
@ngbinh

If I have one single data source (think of a document db that stores JSON documents). Then the data source can be like:

implict object JsonStore extends DataSource[String, JSON]{
  def name: String = ???
  def fetchOne(id: Identity): Query[Option[Result]] = ???
  def fetchMany(ids: NonEmptyList[Identity]): Query[Map[Identity, Result]] = ???
}

Now, we store more than one kinds of document there. Says, User and Post. We also have UserId and PostId that basically wrap a String and enforce certain format for each ids. And UserModel and PostModel are case classes that could be serialized to JSON. What is the right way to have two data sources that return both data types and can batch and cache results for both?

Thanks

Taleb Zeghmi
@talebzeghmi

Hi,
Playing with introduction code http://47deg.github.io/fetch/docs.html#introduction-0

  for {
    x: User <- getUser(1)
  } yield s"${x.id} ${x.username}

Produces following error:
Error:(105, 23) value filter is not a member of fetch.Fetch[User]

Thoughts on how to get around this?

Taleb Zeghmi
@talebzeghmi

got help in cats room:
@emilypi

because for comprehensions are implemented poorly.
every time you have a predicate on your type like : Type or if foo, it calls withFilter

Emily Pillmore
@emilypi
I was pinged in here :P
@tewf what kind of effect are you dealing with that doens’t have withFilter?
Dermot Haughey
@hderms
@raulraja nah I agree I think in retrospect destructive updates don't make sense with it
Patrick Curran
@patrickthebold_twitter
I just started looking at fetch, and I might have a hard time formulating this question.
I like the deduplication feature, but when I run a fetch I'd like to be able to get the data as it becomes available. (Imagine I have a websocket and I want to push data through it as it becomes available.) My issue is that if I use:
(fetchA,fetchB).tupled even if the two fetches ultimately execute in parallel, I have to wait until they both complete to get the full tuple.
I think I want to be able to run the fetch and get an F[(F[A],F[B])] or something like that.
Patrick Curran
@patrickthebold_twitter
So somehow I'd like to get separate ConcurrentEffects that still share the same underlying batching/caching logic.
Alejandro Gómez
@purrgrammer
hey Patrick, your use case is interesting but not something the library supports at the moment
one of the requirements of the library is that data should fit in memory, so it is fine for constructing HTTP responses, but not a good fit for streaming through websockets or server-sent events
Patrick Curran
@patrickthebold_twitter
I'm not sure this changes anything, but in my case memory is not an issue. I could wait until I have everything and then send to my UI, but I'd like to send some data from the first round of calls, render what I can, and then send subsequent data.
Alejandro Gómez
@purrgrammer

I could wait until I have everything and then send to my UI, but I'd like to send some data from the first round of calls, render what I can, and then send subsequent data.

the use case makes sense, although I'm not sure Fetch could support that at the moment. I've been talking with a colleague about how we could support it but I need to spend some more cycles on it to have an answer

i used Fetch for reading data for a UI, although it was pre-rendered in the server
it then runs the Fetch in the client too in case something has been updated
Patrick Curran
@patrickthebold_twitter

thanks, I just wanted to be clear it wasn't a memory/infinite stream thing I was asking for. Thinking out loud: How about attaching an IO action/callback to a Fetch. Something like:

def subFlatMap[A, B](fa: Fetch[F, A])(f: A => F[B]): Fetch[F, B] = Unfetch(fa.run.flatMap {
      case Done(v) => f(v).map(Done(_))
      case Throw(e) =>
        Applicative[F].pure(Throw[F, B](e))
      case Blocked(br, cont) =>
        Applicative[F].pure(Blocked(br, subFlatMap(cont)(f)))
    })

Then you could do:
```
def sendToClientA: F[A] = for {
_ <- sendData(a)
} yield a
def sendDataA: F[Unit] = ???

Anyway I'll keep thinking about it...(And trying to figure out cats-effect) thanks again!
Ryan Tomczik
@Tomczik76
I have a question about how the Cache works. Does it have a time to live?
Alejandro Gómez
@purrgrammer
@Tomczik76 glad you asked. The default implementation of the cache uses an in-memory cache with no TTL, so all the data in a fetch is kept in the cache. Note that using a cache with a TTL could cause different values to be returned for the same identity. Since we want to keep the guarantee that once you read an identity it will be the same for the whole Fetch execution, we use no TTL in the cache.