Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 15:41
    SethTisue commented #12489
  • 15:08
    SethTisue commented #9818
  • 15:08
    SethTisue labeled #9818
  • 15:07
    SethTisue commented #12491
  • 12:30
    joroKr21 commented #12489
  • 10:07
    scala-jenkins milestoned #9818
  • 10:07
    joroKr21 opened #9818
  • Dec 03 21:19
    SethTisue commented #2258
  • Dec 03 21:19

    SethTisue on main

    Fix invalid links to features i… (compare)

  • Dec 03 21:19
    SethTisue closed #2244
  • Dec 03 21:19
    SethTisue closed #2258
  • Dec 03 20:42
    som-snytt commented #10604
  • Dec 03 20:39
    ira19921 commented #10604
  • Dec 03 19:10
    som-snytt commented #10604
  • Dec 03 18:13
    SethTisue demilestoned #11835
  • Dec 03 17:57
    som-snytt commented #9490
  • Dec 03 17:54
    som-snytt closed #11835
  • Dec 03 17:54
    som-snytt commented #11835
  • Dec 03 17:18
    Mr-HHP commented #10604
  • Dec 03 17:15
    er1c commented #10604
Khagay Nagdimov
@khagaydv01
@BalmungSan yes, it's running in a spark application but, at that moment, there is no spark method called
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 I mean, maybe your code looks like this?
def foo(i: Int): String = {
  if (i == 0) throw new CustomException()
  else i.toString
}

def bar(data: Dataset[Int]): Try[Dataset[String]] =
  Try(data.map(foo))
Seth Tisue
@SethTisue
the lampepfl/dotty room has been shut down, so this is now a Scala 2 and 3 room
Luis Miguel Mejía Suárez
@BalmungSan
:open_mouth:
Bjorn Regnell
@bjornregnell
:thumbsup:
Khagay Nagdimov
@khagaydv01
@BalmungSan yes! just like that
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 you need to understand two things:
  1. Try(data.map(foo)) is running in the master node, whereas the execution of foo is running in multiple executor nodes; as such Try won't be able to catch those exceptions.
  2. data.map(foo) didn't do anything at all, it just added a node to the DAG telling the runtime that when running the pipeline to reach data it then needs to run a map(foo) - So, that call didn't throw any exception, the moment the exception is thrown is when you run an action on the resulting Dataset.
You may do data.map(x =>Try(foo(x))) which would catch each individual exception and you would have a Dataset of Try of String.
But I actually do not know if Spark would support that and if it would make it easier to deal with failures.
My gut feeling is that there won't be an Encoder for Try. But, hopefully, I would be wrong.
DavidGoodenough
@DavidGoodenough
Would it be sensible to prefix all remarks with either "Scala-2" or "Scala-3" so that the context is obvious now that the dotty room has merged in here?
Luis Miguel Mejía Suárez
@BalmungSan
Not really a prefix but yeah, it would help if people start to make it clear if their question is Scala 2 specific, Scala 3 specific (as well as with many other categories like sbt, Spark, FP, design, etc)
In any case, I guess most of the time the context would be clear.
Seth Tisue
@SethTisue
Martin's ScalaCon keynote starts in 5 minutes 🍿
D Cameron Mauch
@DCameronMauch
I know this is off topic, but does anyone know how to get a listing of available libraries from a Maven repository? I’m trying to get it from this one: https://mmlspark.azureedge.net/maven
Luis Miguel Mejía Suárez
@BalmungSan
I use https://mvnrepository.com/ but always from the web page
Ah wait, I believe I misunderstood, you mean from a specific (probably private) repository?
D Cameron Mauch
@DCameronMauch
Yes, for the usual repositories, I also use that same link
Adam Rosien
@arosien
i usually go to http://search.maven.org/
D Cameron Mauch
@DCameronMauch
Yeah, that doesn’t show newer versions that I know are on the repo link above
I just don’t know what other versions may be there
Nathan Fischer
@nrktkt:matrix.org
[m]
I know mill has a command to view newer versions of dependencies, so it might be worth digging in there to see how it's doing that
Adam Rosien
@arosien
well, for example if you are looking for versions of cats-effect, you can go to https://search.maven.org/artifact/org.typelevel/cats-effect_2.13 and it shows you all the versions. the pages are all hyperlinked so you can navigate around to find stuff
Nathan Fischer
@nrktkt:matrix.org
[m]
@arosien: that's just for maven central, I believe he's looking for a library that's not on central
looks like azure?
Luis Miguel Mejía Suárez
@BalmungSan
random (off-topic?) thought; I haven't seen Gavin over here for a while, does anyone know if he is ok?
Dermot Haughey
@hderms
have you guys ever seen while loops be slower than .foreach
I'm getting a while loop over the indices of some string to be consistently slower than s.indices.foreach{i =>}
I really have no compelling explanation for why that's the case given that no matter how well JIT'd a foreach loop could be I don't see how it could compile to something faster than a while loop
Nathan Fischer
@nrktkt:matrix.org
[m]
sounds odd, what does the while loop look like?
Dermot Haughey
@hderms
basically exactly what you'd expect
var i = 0 while (i < s.length) { //same body as foreach i += 1 }
not sure why gitter collapsed that to one line
Matthew Rooney
@javax-swing
@DavidGoodenough what is your collection type? the s.length may be calculated each time
nrktkt
@nrktkt:matrix.org
[m]
I had that thought, but sounds like it's a string
Dermot Haughey
@hderms
yeah it's a string
Matthew Rooney
@javax-swing
o yeah i missed that... bizarre
Dermot Haughey
@hderms
this isn't a super scientific comparison so it could have just been random chance but it's still surprising given I had a bunch of consistent runs
was just curious if there was something i was somehow missing
I know in rust for char in string.chars() can be considerably faster than using indices because it can eliminate some bounds checking and things like that
Dermot Haughey
@hderms
it's possible that JVM optimization of lamdbas is good enough to not only transform a first class function invocation down to something very machine-friendly but also use information about what's going on to also eliminate bounds checking
Jens
@DieBauer
I’m playing around with Scala3, but can’t seem to find the replacement of TypeTag, in construct like: class MyClass[A: TypeTag](a: A)
1 reply
Nathan Fischer
@nrktkt:matrix.org
[m]
@hderms: how much faster/slower are you seeing? if it seems like a lot, the next step would be to drop an example in JMH to confirm
Seth Tisue
@SethTisue
@DieBauer what are you using TypeTag for? what paths forward are available depend on that. see e.g. lampepfl/dotty-feature-requests#62
Luis Miguel Mejía Suárez
@BalmungSan
@hderms that is very weird considering that foreach in String is implemented using a while: https://github.com/scala/scala/blob/v2.13.6/src/library/scala/collection/StringOps.scala#L1035
They do cache the s.length in a variable but I do not think that should matter a lot, or does it?
When I need a while I also usually cache all that kind of stuff just to be sure.
But still, I agree it would be good to do a proper benchmark.
Martijn
@martijnhoekstra:matrix.org
[m]
Maybe it does, the jit sometimes knows some bound patterns that it eliminates
Oh, that would then be the other way around
Still might be it.
Dermot Haughey
@hderms
I'm going to write a micro benchmark of some kind I think