Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 21 22:50
    peter-nuernberg synchronize #2207
  • Oct 21 18:15
    SethTisue milestoned #12474
  • Oct 21 18:15
    SethTisue demilestoned #12474
  • Oct 21 18:15
    SethTisue demilestoned #12470
  • Oct 21 18:15
    SethTisue milestoned #12470
  • Oct 21 18:15
    SethTisue milestoned #12463
  • Oct 21 18:15
    SethTisue demilestoned #12463
  • Oct 21 18:15
    SethTisue milestoned #12438
  • Oct 21 18:15
    SethTisue demilestoned #12438
  • Oct 21 18:15
    SethTisue milestoned #12437
  • Oct 21 18:15
    SethTisue demilestoned #12437
  • Oct 21 18:15
    SethTisue milestoned #12418
  • Oct 21 18:15
    SethTisue demilestoned #12418
  • Oct 21 18:15
    SethTisue milestoned #12378
  • Oct 21 18:15
    SethTisue demilestoned #12378
  • Oct 21 18:15
    SethTisue demilestoned #12340
  • Oct 21 18:15
    SethTisue milestoned #12340
  • Oct 21 18:15
    SethTisue milestoned #12339
  • Oct 21 18:15
    SethTisue demilestoned #12339
  • Oct 21 18:15
    SethTisue milestoned #12275
Geoffrey Knauth
@gknauth
I mentioned the recent release of Scala 3, and a coworker said Databricks has only recently started supporting Scala 2.12. Is that correct? I’m trying to figure out the timeline to being able to use Scala 3 with Databricks.
Luis Miguel Mejía Suárez
@BalmungSan
Disclaimer I have no relationship with them, nor with the compiler team. And I am just a random person in the Internet with some resentments about Spark. So do not take me too serious.
Seth Tisue
@SethTisue
@gknauth Spark has been on 2.12 for a long time. they're close to shipping 2.13 support: https://issues.apache.org/jira/browse/SPARK-25075 . and there are comments on the same ticket talking about the 3 situation
Luis Miguel Mejía Suárez
@BalmungSan
But I guess Spark won't support Scala 3 until 2 years. I hope I am wrong.
Geoffrey Knauth
@gknauth
Thanks @SethTisue
Seth Tisue
@SethTisue
2 years sounds like a reasonable guess
Geoffrey Knauth
@gknauth
Thanks also @BalmungSan , good to clear the pond of rocks before jumping in.
Khagay Nagdimov
@khagaydv01
Hello. I'm encountering an interesting situation. I have a method that is wrapped in a Try. That method has a sub-method that throws an exception but, the return value of that sub-method is a String. It seems that the exception gets thrown and the thread crashes even though the parent method is wrapped in a Try. Is that expected behavior? I would have thought the exception gets wrapped in a Failure.
Jim Newton
@jimka2001
Is there a better idiom for following? I.e., without interrupting the exception being thrown, take a different finally action based on whether the expression finished or aborted? I'm using a mutable variable.
      println("[ starting " + testName)
      var finished = false
      try{
        testFun
        finished = true
      }
      finally{
        if (finished)
          println("] finished " + testName)
        else
          println("] aborted" + testName)
      }
Seth Tisue
@SethTisue
@khagaydv01 what kind of exception is it? Try distinguishes fatal from nonfatal exceptions and only catches the nonfatal ones
Khagay Nagdimov
@khagaydv01
@SethTisue a custom exception that derived from RuntimeException
Jim Newton
@jimka2001
in this particular case it is a java.lang.OutOfMemoryError: Java heap space
keep in mind I don't want the exception tampered with. I still want it to fail, I just want to indicate that the test case finished successfully or that it aborted
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 can you create a MCVE in Scastie?
That should work.
@jimka2001 AFAIK the moment you have a fatal exception like a memory error you can't do anything at all.
But I may be wrong.
Khagay Nagdimov
@khagaydv01
weird thing is when i throw that same exception in the method that is wrapped a Try, it gets wrapped in a Failure
seems like sub-methods that throw exception that aren't wrapped in a Try crash the thread
even if the parent method is wrapped in a Try
Walter Chang
@weihsiu

is it possible to make an extension of a f-bounded type?

object FBounded:
  trait Self[A <: Self[A]]:
    def bar(): Unit = ()

  extension [A <: Self[A]](self: A)
    def foo(): Unit = ()

  trait Me extends Self[Me]

  val self: Self[Me] = new Me {}
  self.bar()
  self.foo() // fails to compile

this last line failed to compile:

[error] 12 |  self.foo()
[error]    |  ^^^^^^^^
[error]    |  value foo is not a member of FBounded.Self[FBounded.Me].
[error]    |  An extension method was tried, but could not be fully constructed:
[error]    |
[error]    |      FBounded.foo[A](self)
[error] one error found
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 is that sub method run in another thread or is Spark related?
@weihsiu it works if you leave its type as Me: https://scastie.scala-lang.org/BalmungSan/IGf7rvujQZmupUSma2i3EA/1
Which makes sense Self[Me] is not a A <: Self[A]
Walter Chang
@weihsiu
@BalmungSan ah, thanks!!
Khagay Nagdimov
@khagaydv01
@BalmungSan yes, it's running in a spark application but, at that moment, there is no spark method called
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 I mean, maybe your code looks like this?
def foo(i: Int): String = {
  if (i == 0) throw new CustomException()
  else i.toString
}

def bar(data: Dataset[Int]): Try[Dataset[String]] =
  Try(data.map(foo))
Seth Tisue
@SethTisue
the lampepfl/dotty room has been shut down, so this is now a Scala 2 and 3 room
Luis Miguel Mejía Suárez
@BalmungSan
:open_mouth:
Bjorn Regnell
@bjornregnell
:thumbsup:
Khagay Nagdimov
@khagaydv01
@BalmungSan yes! just like that
Luis Miguel Mejía Suárez
@BalmungSan
@khagaydv01 you need to understand two things:
  1. Try(data.map(foo)) is running in the master node, whereas the execution of foo is running in multiple executor nodes; as such Try won't be able to catch those exceptions.
  2. data.map(foo) didn't do anything at all, it just added a node to the DAG telling the runtime that when running the pipeline to reach data it then needs to run a map(foo) - So, that call didn't throw any exception, the moment the exception is thrown is when you run an action on the resulting Dataset.
You may do data.map(x =>Try(foo(x))) which would catch each individual exception and you would have a Dataset of Try of String.
But I actually do not know if Spark would support that and if it would make it easier to deal with failures.
My gut feeling is that there won't be an Encoder for Try. But, hopefully, I would be wrong.
DavidGoodenough
@DavidGoodenough
Would it be sensible to prefix all remarks with either "Scala-2" or "Scala-3" so that the context is obvious now that the dotty room has merged in here?
Luis Miguel Mejía Suárez
@BalmungSan
Not really a prefix but yeah, it would help if people start to make it clear if their question is Scala 2 specific, Scala 3 specific (as well as with many other categories like sbt, Spark, FP, design, etc)
In any case, I guess most of the time the context would be clear.
Seth Tisue
@SethTisue
Martin's ScalaCon keynote starts in 5 minutes 🍿
D Cameron Mauch
@DCameronMauch
I know this is off topic, but does anyone know how to get a listing of available libraries from a Maven repository? I’m trying to get it from this one: https://mmlspark.azureedge.net/maven
Luis Miguel Mejía Suárez
@BalmungSan
I use https://mvnrepository.com/ but always from the web page
Ah wait, I believe I misunderstood, you mean from a specific (probably private) repository?
D Cameron Mauch
@DCameronMauch
Yes, for the usual repositories, I also use that same link
Adam Rosien
@arosien
i usually go to http://search.maven.org/
D Cameron Mauch
@DCameronMauch
Yeah, that doesn’t show newer versions that I know are on the repo link above
I just don’t know what other versions may be there
Nathan Fischer
@nrktkt:matrix.org
[m]
I know mill has a command to view newer versions of dependencies, so it might be worth digging in there to see how it's doing that
Adam Rosien
@arosien
well, for example if you are looking for versions of cats-effect, you can go to https://search.maven.org/artifact/org.typelevel/cats-effect_2.13 and it shows you all the versions. the pages are all hyperlinked so you can navigate around to find stuff
Nathan Fischer
@nrktkt:matrix.org
[m]
@arosien: that's just for maven central, I believe he's looking for a library that's not on central
looks like azure?
Luis Miguel Mejía Suárez
@BalmungSan
random (off-topic?) thought; I haven't seen Gavin over here for a while, does anyone know if he is ok?
Dermot Haughey
@hderms
have you guys ever seen while loops be slower than .foreach
I'm getting a while loop over the indices of some string to be consistently slower than s.indices.foreach{i =>}