we have a query for a bunch of IDs, and for each of those we have to read some data, so some work, and then write some data back out. We do the batching manually right now (usually as one big batch or all individual requests) but fetch’s batch/parallelism while letting the functions look like their operating on one entity at a time is pretty appealing
But I guess these are special cases of writes, which are idempotent and do not affect any of the reads
I was also wondering about the utility of a Put style version of Fetch. The interesting part for me would be combining Put and Fetch jobs together and have Put be able to update the cache that the Fetch jobs are using. But as @peterneyens says there would be a lot of edge cases to think about
Exception in thread "main" java.lang.NoClassDefFoundError: cats/FunctorFilter
What I missed?
This is my dependencies:
libraryDependencies += "org.typelevel" %% "cats-core" % "1.0.0-MF"
libraryDependencies += "com.47deg" %% "fetch" % "0.6.3"
@a-nigredo Fetch still depends on
cats 0.9.0 which had the FunctorFilter there. We are currently working towards moving it to 1.0.0-MF and FunctorFilter has moved to FunctorEmpty in a new lib cats-mtl. @peterneyens do you know how close are we to release Fetch for cats 1.0.0-MF?
@peterneyens Looks like Alex recently addressed that https://github.com/monix/monix/releases/tag/v3.0.0-M1
i am new to fetch, and i will be glad if someone can help a few issues:
- can i create an api which is not part of the DataSource trait, and call it ?
- can i define an api which save/update and not just fetching data ?
@eranmiz132
The only way Fetch can reach out to external sources is through the fetchOne and fetchMany methods of DataSource.
Can you give an example what you want to accomplish?
For the second point, there was some discussion in the posts above about just that (starting at https://gitter.im/47deg/fetch?at=599de37042d2f61025897618).
Summary: For the moment Fetch only supports fetching data
thank u for the reply ! well, i have a lot of different api which get items, but the for different keys. if i extends the DataSource trait, i can implement only one "kind" of get by key, or i need a lot of classes to extends the DataSource, each with different key. it is not convenient.
the branch itself seems in pretty good shape, I was able to upgrade code without incendent
Hi! I made this side project while I was on vacation I was inspired by your issue #11 (visualize fetch rounds with reftree). There is a blog post and github repo http://justinhj.github.io/2017/10/11/hacker-news-api-3.html
The reftree visualization looks quite nice indeed.
I'm using https://github.com/etaty/rediscala which should in theory but I don't know if it allows pipelining what you want specifically as I haven't used the cluster support yet.
I am Aarsh , a Scala dev looking to contribute to some open source projetcs
Is it possible for me to work on Fetch
Considering that it's been developed by 47 degrees ?
Also, any list of planned features / issues/ bug fixes I can work on to get my foot in the door ? Would really appreciate the help.
Java has no way to represent scala higher kinds and some of the methods would need to be made concrete or specialized. Same for the implicit resolution would have to be wrapped in a scala project that exposes concrete types and already resolved instances with signatures compatible in Java. If you plan on going down that rabbit hole we'd welcome such a submodule in Fetch itself
You probably want
ListenableFuture<A> as result type from guava, we have instances to go to that and back in Freestyle async so a Fetch in terms of scala.concurrent.Future can just be ~> to guava's ListenableFuturepotentially.