Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
nabih.nebbache
@ThegreatShible
I did that
trait Reduce[F[_]] {
def reduceA(f : (A,A) => A) : A
}
Rob Norris
@tpolecat
Yeah.
nabih.nebbache
@ThegreatShible
With that I can reduce on any collection, but i was wondering whether there is an already existant data type like that in cats
Rob Norris
@tpolecat
If you're using Cats then Reducible will do what you want but I suspect you can't define instances for RDD and so on.
nabih.nebbache
@ThegreatShible
yes that the problem
Reducible contains reduceleft and reduceright that are not existant in RDDs (or any parallel collection)
Rob Norris
@tpolecat
Oh I see.
nabih.nebbache
@ThegreatShible
So I think I am going with my own Trait
Rob Norris
@tpolecat
Yeah.
Gavin Bisesi
@Daenyth
Also if you want it to be appropriate for RDD/par collections you need that CommutativeMonoid (or at least CommutativeSemigroup) constraint
Rob Norris
@tpolecat
It doesn't need to be commutative unless reduce scrambles the elements.
It does need to be associative though.
Gavin Bisesi
@Daenyth
par one would, no?
and rdd?
nabih.nebbache
@ThegreatShible
No only associative, no need for commutative
Rob Norris
@tpolecat
As long as parallel reduce combines the chunks in order it only needs to be associative. If it jumbles them up then yeah commutative.
Ethan
@esuntag
The Spark documentation (at least for the latest version) specifies that reduce needs to be commutative and associative
I think reduce is allowed to force an arbitrary shuffle
Rob Norris
@tpolecat
Yeesh.
Ethan
@esuntag
I might be misremembering on the shuffle, it might just be reduceByKey that will force it. Either way, I don't think the elements in an RDD are guaranteed in any specific order
You can maybe right a custom reader/partitioner that can try to provide those guarantees, but you'll be much better off just asking for a CommutativeMonoid
Yuriy Badalyantc
@LMnet
Hi! Is there any specific reason why there is no Eq[Array[T]] instance?
nigredo-tori
@nigredo-tori
@LMnet, Array is a mutable structure, so it's not referentially transparent, and collection equality makes little sense. E.g. if you had such instance, you could have two arrays be "equal" at some point in time, and "not equal" at another.
Yuriy Badalyantc
@LMnet
Ok, thanks. Completely forgot about arrays mutability :)
Gavin Bisesi
@Daenyth
If you want something array-like (especially for say, byte array performance), you might consider using fs2.Chunk
Gavin Bisesi
@Daenyth
Can this be merged and shipped?
Artem Egorkine
@arteme

Whever I use EitherT.right(...) in a for comprehension I get errors like:

type mismatch;
 found   : Int => cats.data.EitherT[scala.concurrent.Future,Nothing,Unit]
 required: Int => cats.data.EitherT[scala.concurrent.Future,cats.data.NonEmptyList[String],Unit]

I have to manually do EitherT.right[NonEmptyList[String]](...)... Is there a way to make the scala compiler automatically deduce the correct type? What am I missing?

Adam Rosien
@arosien
@arteme if you add a type parameter for the "left" type, it will produce the correct output type, aka EitherT.right[L](...)
Artem Egorkine
@arteme

@arosien I know. Consider code:

def v1: Future[ValidatedNel[String, Int]] = Future.successful(1.valid)
def f1: Future[Int] = Future.successful(2)

val n = for {
  _ <- v1.asEitherT
  int <- EitherT.right[NonEmptyList[String]](f1)
} yield {
  int + 1
}

If I already have an EitherT[Future, NonEmptyList[String], Int] in there I'd imagine the compile can guess the proper "left" type for Either.right call.

For clarity:
implicit class ValidatedFuture[A, B](val e: Future[A Validated B]) extends AnyVal {
  def asEitherT(implicit ec: ExecutionContext): EitherT[Future, A, B] = EitherT(e map { _.toEither })
}
Adam Rosien
@arosien
ah sorry, i missed the important part you mentioned. i hurt my back and can't quite see straight.
usually there's some issue that makes the compiler not as smart as us humans, and the normal answer is that scala 3 will fix this :(
Adam Rosien
@arosien
@arteme if you annotate the type of val n the error type will infer correctly:
val n: EitherT[Future, NonEmptyList[String], Int] = for {
  _ <- v1.asEitherT
  int <- EitherT.right(f1)
} yield {
  int + 1
}
the issue is the inference uses the first value of the for, but the asEitherT has ambiguous type parameters without help
nafg
@nafg
Is there an operator that with the right typeclass tuples two things -- (F[A], F[B]) => F[(A, B)] ?
nafg
@nafg
I mean is there an operator F[A] op F[B]
Oh I see (a, b).tupled
Fabio Labella
@SystemFw
fa product fb as well
tupled works for tuples of any size
Gavin Bisesi
@Daenyth
the tuple form is great - especially with mapN or parMapN instead
Daniel Capo Sobral
@dcsobral
Is there some A => (A => B) => (A => C) => (B, C)-like method/implicit syntax?
Rob Norris
@tpolecat
If you have f: A => B and g: A => C then f product g is A => (B, C)
Daniel Capo Sobral
@dcsobral
Cool.
Rob Norris
@tpolecat
Or (f, g).tupled more generally .. or (f, g).mapN(...) yet more generally.
(A => ? is the reader monad so all the applicative composition stuff works.)
Daniel Capo Sobral
@dcsobral
Though, sadly, that order doesn’t really help me, since it’s just two different methods I’m calling on the same object.
Ah, well, it’s just a few characters.