Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 28 23:23
    pweisenburger commented #12498
  • Nov 28 23:00
    som-snytt synchronize #9814
  • Nov 28 23:00
    som-snytt commented #9814
  • Nov 28 20:56
    som-snytt edited #9814
  • Nov 28 20:55
    som-snytt synchronize #9814
  • Nov 28 20:30
    som-snytt closed #10276
  • Nov 28 20:30
    som-snytt commented #10276
  • Nov 28 20:03
    ritschwumm commented #10276
  • Nov 28 17:42
    som-snytt commented #10276
  • Nov 28 17:41
    som-snytt commented #10276
  • Nov 28 14:42
    IowU opened #2239
  • Nov 28 14:09
    joroKr21 assigned #12493
  • Nov 28 14:09
    joroKr21 unlabeled #12493
  • Nov 28 14:09
    joroKr21 review_requested #9815
  • Nov 28 14:09
    joroKr21 ready_for_review #9815
  • Nov 28 13:51
    arata-honda synchronize #2235
  • Nov 28 11:35
    joroKr21 commented #10276
  • Nov 28 11:32
    joroKr21 commented #10276
  • Nov 28 11:19
    joroKr21 synchronize #9815
  • Nov 28 09:38
    scharom16 closed #2238
felher
@felher

Re monad transformers:

I think most of the time it doesn't matter. Especially if you consider ZIO directly which means you are probably in the domain of normal "real world code".

On the other hand, for the more unconstrained question of "are monad transformers slow?", I think there are real-world examples where it might matter. If you are rolling your own parsers and are basing them on something like ReaderT[Env, WriterT[Log, StateT[Pos, ...]]], I would actually expect that the performance drop could matter here.

crater2150
@crater2150:qwertyuiop.de
[m]
Ah, SortedSet seems to manage by returning a SortedSet from methods, if an Ordering is available, otherwise a Set is returned. But it looks like this requires implementing all such methods manually for my typeclass.
Rob Norris
@tpolecat
@felher right, there are cases where you do want to optimize them away but that's not the use case they're talking about
Luis Miguel Mejía Suárez
@BalmungSan
@crater2150:qwertyuiop.de does it need to be a "real collection"? Like you want to pass it where some arbitrary Iterable is expected? Otherwise maybe it would be easier to just provide it as a custom data structure. (or maybe you are taking advantage of the generated methods)
crater2150
@crater2150:qwertyuiop.de
[m]
@BalmungSan: It already works as an Iterable, but the generated methods (like map, flatMap,...) return an Iterable, instead of the narrower type. But I just found this list of methods that need to be overridden manually for this purpose
With 2.12, the CanBuildFrom stuff apparently took care of that
felher
@felher
@tpolecat Not disagreeing, just wasn't completely sure what part exactly they are talking about. If they were only talking about ZIO, I completely agree. When talking about ZPure (not ZIO, but the same eco system), I think there is more nuance there because the default use cases will be very different. Of course, cats has it's own IndexedReaderWriterStateT which you could use peel the transformer layers off.
@crater2150:qwertyuiop.de Is this your matrix/gitter integration?
crater2150
@crater2150:qwertyuiop.de
[m]
@felher: that's provided by gitter itself
felher
@felher
@crater2150:qwertyuiop.de ah, nice
Luis Miguel Mejía Suárez
@BalmungSan
@crater2150:qwertyuiop.de my point was mostly if you are wining something by making it a Iterable if so, hope you can manage to fix the problem - if not (or maybe just too little) maybe it would be better to just not try to do that
crater2150
@crater2150:qwertyuiop.de
[m]
well, it looks like I have to add overloads for some functions that I need, so I just get a bit less stuff automatically than with 2.12. But they're all methods that I would have to write manually too if it wasn't an Iterable 🤷‍♂️
Rob Norris
@tpolecat
@BalmungSan +1 Not doing things is a winning strategy.
Luis Miguel Mejía Suárez
@BalmungSan
@crater2150:qwertyuiop.de makes sense, good luck with that. Did you also mixed-in the Ops trait? I believe that one is the one that provides most stuff out of the box.
crater2150
@crater2150:qwertyuiop.de
[m]
The Ops trait was the one that gave me problems, because it requires constructors without additional type class constraints on the element type
But it looks like most methods that require overloading can be done with from(new View.Map(...)) etc.
Martijn
@martijnhoekstra:matrix.org
[m]
Being a real collection IMO is much like Pinochio who wants to be a real boy. In the end, the question whether some implementation is a real collection or not is more about the definition of the real collection than about the implementation
and the real collection was the friends we made along the way
which we added to the collection
Rob Norris
@tpolecat
i feel like you're starting to channel som snytt
Martijn
@martijnhoekstra:matrix.org
[m]
I may only dream on one day reaching their insightful / understandable ratio
Henri Cook
@henricook
Hi all, i'm a noob to scalamock, i've been a Mockito guy for years. I'm trying to convert a Mockito unit test to a scalamock one and i can't see how to mock a val of a class - can anyone help me out? I feel like it's probably a really simple question but I can't see it anywhere in the docs
e.g. I have class Foo { val aThing: AObject, val bThing: BObject, val cThing: CObject } and I want a mock[Foo] where I can set the value of aThing only
Rohan Sircar
@rohan-sircar

and the real collection was the friends we made along the way
which we added to the collection

aha!

Jim Newton
@jimka2001
I have a set of test suites written using scalatest. When I run all the tests (at least from within IntelliJ), the tests fail for lack of heap space. However, when I run each of the tests in isolation, there is no out of heap error. I'd love to hear suggestions about how to solve this problem?
Rob Norris
@tpolecat
Have you tried increasing your heap space? The default isn't very much.
-Xmx<size> like -Xmx8000m
actually I think you can say -Xmx8g now
Jim Newton
@jimka2001
Sorry, but I don't know where to put -Xmx8g at
Rob Norris
@tpolecat
On the commandline this is what you do. I don't know how to tell IntelliJ to do it. My guess is that it's in the run configuration for your tests.
Someone here who uses IJ can tell you.
D Cameron Mauch
@DCameronMauch
I’m trying to figure out how to create a method with this signature: def fields[T](): List[String] where the output is a list of the field names in a case class T
Rob Norris
@tpolecat
You can do that with shapeless.
D Cameron Mauch
@DCameronMauch
@jimka2001 “IntelliJ IDEA” -> “preferences” -> “build, execute, deploy” -> “compiler” -> “scala compiler” -> “scala compile server"
Jim Newton
@jimka2001
I can add -Xmx8g to the VM options? beginning? end? doesn't matter?
Rob Norris
@tpolecat
Doesn't matter.
D Cameron Mauch
@DCameronMauch
There is the setting right there for maximum heap size
I just change that
Thanks for the pointer
Rob Norris
@tpolecat
In Scala 3 you can do it with Mirror but for Scala 2 you need Shapeless.
An awful lot of questions here (and elsewhere) are about things that are at the edge of what the language can do, and these are exactly the kinds of things that changed in Scala 3. So I think a lot of questions are going to have two answers for a while.
D Cameron Mauch
@DCameronMauch
This seems to work:
implicit class DatasetOps[T: Encoder](ds: Dataset[T]) {
    def asCleaned[U: Encoder](): Dataset[U] = {
      ds.select(
        classOf[U]
          .getDeclaredFields
          .toList
          .map(_.getName)
          .map(col): _*
      ).as[U]
    }
}
Spark is weird. If you have a DataFrame with 10 columns, and convert it to a Dataset of some case class with 6 fields, those extra 4 column are still there. Taking space, slowing down shuffles, etc. This is my attempt at removing all the crud.
D Cameron Mauch
@DCameronMauch
Ah, looks like the above doesn’t compile, though IntelliJ is not showing any errors
Rob Norris
@tpolecat
The declared fields are not necessarily the same thing as the primary constructor arguments.
Which I assume is what you meant by "fields"
D Cameron Mauch
@DCameronMauch
This compiles:
    def asCleaned[U: ClassTag: Encoder](): Dataset[U] = {
      val fields: List[Column] = implicitly[ClassTag[U]].runtimeClass.getDeclaredFields.toList.map(_.getName).map(col)
      ds.select(fields: _*).as[U]
    }
1 reply
It seems to generate the expected list, except in DataBricks, which add some $outer fields to the end...
I’m not sure I understand the difference