Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Feb 01 05:14

    47erbot on scala3-library-3.2.2

    (compare)

  • Feb 01 05:14

    47erbot on main

    Update scala3-library to 3.2.2 (compare)

  • Feb 01 05:14
    47erbot closed #697
  • Feb 01 05:08
    47erbot opened #697
  • Feb 01 05:08

    47erbot on scala3-library-3.2.2

    Update scala3-library to 3.2.2 (compare)

  • Jan 30 05:15

    47erbot on mdoc-2.3.7

    (compare)

  • Jan 30 05:15

    47erbot on main

    Update mdoc, sbt-mdoc to 2.3.7 (compare)

  • Jan 30 05:15
    47erbot closed #696
  • Jan 30 05:09
    47erbot opened #696
  • Jan 30 05:09

    47erbot on mdoc-2.3.7

    Update mdoc, sbt-mdoc to 2.3.7 (compare)

  • Jan 27 05:15

    47erbot on sbt-scalajs-1.13.0

    (compare)

  • Jan 27 05:15
    47erbot closed #695
  • Jan 27 05:15

    47erbot on main

    Update sbt-scalajs, scalajs-com… (compare)

  • Jan 27 05:07
    47erbot opened #695
  • Jan 27 05:07

    47erbot on sbt-scalajs-1.13.0

    Update sbt-scalajs, scalajs-com… (compare)

  • Jan 25 05:15

    47erbot on scalafmt-core-3.7.1

    (compare)

  • Jan 25 05:15
    47erbot closed #694
  • Jan 25 05:15

    47erbot on main

    Update scalafmt-core to 3.7.1 (compare)

  • Jan 25 05:08
    47erbot opened #694
  • Jan 25 05:08

    47erbot on scalafmt-core-3.7.1

    Update scalafmt-core to 3.7.1 (compare)

Tristan Lohman
@gatorcse
Is there similar library to this for posting data?
Just looking at the API, could this be used to send/store data as well as fetch it?
DataSource[I, A], with I as your object type and A as your datasource api result type (DBResult or whatever)
apologies if this has been asked before
Raúl Raja Martínez
@raulraja
@gatorcse I think the value of Fetch is in it's optimizations for deduplication, batching and parallelism. If we are talking about posting data and using something like Fetch or Fetch itself we would have to determine what those mean in the context of posting.
  • Can we batch posts? that is probably doable in the same way we use datasources today
  • Can we deduplicate posts? Maybe if once one is acknowledged posts with the same data to the same id are ignored.
  • Can we paralelize posts? As long as the semantics are of independent operations and the value does not matter and the order in which they happen does not affect the result of the program then yes.
    So, yes, something for posting data could potentially be implemented as part of fetch or companion lib and would allow users not to worry about keeping track of the ops they perform to manually not do them again, batch them explicitly or run them in parallel.
Peter Neyens
@peterneyens

The benefits aren't as big as when fetching data I think.

The main benefit with Fetch (and other libraries inspired by the Haxl paper) is that you can just specify the data you need to use without thinking how to get it (efficiently).
If part A, part B and part C all independently need foo you don't need to think about where to get foo so you only request it once.
If parts A, B and Care combined in one Fetch[Result], foo will automatically be fetched only once.

Posting data can also be a lot more complex because posts can influence each other.
For example what do you do with two parallel posts with the same id but different data?

I am interested to know if you have a use case in mind or if it is just a general question.

A quote from the Haxl paper :

When the business logic is only concerned with reading data
from external sources and not writing, the programmer doesn’t
care about the order in which data accesses happen, since there
are no side-effects that could make the result different when the
order changes. So in this case the programmer would be entirely
happy with not having to specify either ordering or concurrency,
and letting the system perform data access in the most efficient way
possible.

Tristan Lohman
@gatorcse
My use case was in some light read-transform-write batch jobs we run
we have a query for a bunch of IDs, and for each of those we have to read some data, so some work, and then write some data back out. We do the batching manually right now (usually as one big batch or all individual requests) but fetch’s batch/parallelism while letting the functions look like their operating on one entity at a time is pretty appealing
But I guess these are special cases of writes, which are idempotent and do not affect any of the reads
Justin Heyes-Jones
@justinhj
I was also wondering about the utility of a Put style version of Fetch. The interesting part for me would be combining Put and Fetch jobs together and have Put be able to update the cache that the Fetch jobs are using. But as @peterneyens says there would be a lot of edge cases to think about
Raúl Raja Martínez
@raulraja
@gatorcse I still think it's a good idea and if it's explicit enough so that the user understands the behavior it's something we could have in Fetch as a separate companion module
Andrey Ivanov
@a-nigredo
Hi all, I try to use fetch "0.6.3" with cats "1.0.0-MF" and in run-time I see an exception
Exception in thread "main" java.lang.NoClassDefFoundError: cats/FunctorFilter
What I missed?
This is my dependencies:
libraryDependencies += "org.typelevel" %% "cats-core" % "1.0.0-MF"
libraryDependencies += "com.47deg" %% "fetch" % "0.6.3"
Raúl Raja Martínez
@raulraja
@a-nigredo Fetch still depends on cats 0.9.0 which had the FunctorFilter there. We are currently working towards moving it to 1.0.0-MF and FunctorFilter has moved to FunctorEmpty in a new lib cats-mtl. @peterneyens do you know how close are we to release Fetch for cats 1.0.0-MF?
Peter Neyens
@peterneyens
We would need a Monix release using cats 1.0.0-MF.
Raúl Raja Martínez
@raulraja
@peterneyens Looks like Alex recently addressed that https://github.com/monix/monix/releases/tag/v3.0.0-M1
eranmiz132
@eranmiz132
hi guys
i am new to fetch, and i will be glad if someone can help a few issues:
  1. can i create an api which is not part of the DataSource trait, and call it ?
  1. can i define an api which save/update and not just fetching data ?
Peter Neyens
@peterneyens

@eranmiz132
The only way Fetch can reach out to external sources is through the fetchOne and fetchMany methods of DataSource.
Can you give an example what you want to accomplish?

For the second point, there was some discussion in the posts above about just that (starting at https://gitter.im/47deg/fetch?at=599de37042d2f61025897618).
Summary: For the moment Fetch only supports fetching data

eranmiz132
@eranmiz132
thank u for the reply ! well, i have a lot of different api which get items, but the for different keys. if i extends the DataSource trait, i can implement only one "kind" of get by key, or i need a lot of classes to extends the DataSource, each with different key. it is not convenient.
chris vale
@crispywalrus
@peterneyens I opened a (trivial) PR against your 1mf branch that changes the name of the twitter module from twitter to fetch-twitter
the branch itself seems in pretty good shape, I was able to upgrade code without incendent
Peter Neyens
@peterneyens
I missed that it was for my branch. Thanks for the new PR anyway.
chris vale
@crispywalrus
@peterneyens np!
Peter Neyens
@peterneyens
@/all We just released Fetch v0.7.0 https://twitter.com/47deg/status/916369464161128449
Raúl Raja Martínez
@raulraja
:clap:
Justin Heyes-Jones
@justinhj
Hi! I made this side project while I was on vacation I was inspired by your issue #11 (visualize fetch rounds with reftree). There is a blog post and github repo http://justinhj.github.io/2017/10/11/hacker-news-api-3.html
Peter Neyens
@peterneyens
@justinhj Nice!
The reftree visualization looks quite nice indeed.
Justin Heyes-Jones
@justinhj
Yeah it's a neat library
Raúl Raja Martínez
@raulraja
@justinhj awesome post and project!
Justin Heyes-Jones
@justinhj
Thanks!
Taleb Zeghmi
@talebzeghmi
Does anyone use the Fetch API in production?
Tom Adams
@tomjadams
Yes. Have done for months.
Taleb Zeghmi
@talebzeghmi
Has anybody found a Redis Scala API that supports Cluster pipelining?
Justin Heyes-Jones
@justinhj
I'm using https://github.com/etaty/rediscala which should in theory but I don't know if it allows pipelining what you want specifically as I haven't used the cluster support yet.
Taleb Zeghmi
@talebzeghmi
rediscala purports to support pipeline but uses transactions
Aarsh Shah
@aarshkshah1992
Hello Guys
I am Aarsh , a Scala dev looking to contribute to some open source projetcs
Is it possible for me to work on Fetch
Considering that it's been developed by 47 degrees ?
Raúl Raja Martínez
@raulraja
@aarshkshah1992 of course! your contributions will be very welcomed!
Aarsh Shah
@aarshkshah1992
Thanks for the reply . Is there a contributing guide I can look into ?
Also, any list of planned features / issues/ bug fixes I can work on to get my foot in the door ? Would really appreciate the help.
Raúl Raja Martínez
@raulraja
no guide at the moment since it's a self contained and small library, feel free to glance over the issues and pick up any of the current open ones and ask any questions here :)