by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 10:41
    diesalbla commented #12072
  • 09:52
    y6x2digc commented #10502
  • 08:28
    ex0ns commented #12070
  • Jul 09 18:19
    SethTisue commented #9108
  • Jul 09 15:34
    som-snytt commented #9108
  • Jul 09 15:27
    som-snytt commented #9106
  • Jul 09 15:11
    lrytz synchronize #9108
  • Jul 09 15:08
    scala-jenkins milestoned #9108
  • Jul 09 15:08
    lrytz opened #9108
  • Jul 09 14:09

    lrytz on pr8865

    min (compare)

  • Jul 09 14:02

    lrytz on 2.13.x

    Improve assertThrows, use it in… Fix checkHasElement to follow a… Merge pull request #9106 from s… (compare)

  • Jul 09 14:02
    lrytz closed #9106
  • Jul 09 13:34
    SethTisue demilestoned #9103
  • Jul 09 11:49
    SethTisue synchronize #1172
  • Jul 09 07:39
    lrytz synchronize #9106
  • Jul 09 06:46
    NthPortal labeled #9107
  • Jul 09 06:46
    NthPortal labeled #9107
  • Jul 09 05:44

    lrytz on pr8865

    (compare)

  • Jul 09 01:35
    szeiger commented #9107
  • Jul 09 01:34

    szeiger on indexed-sliding

    (compare)

Leif Warner
@LeifW
What version of spark are you on?
Oh, 2.1.1?
Oğuzhan Kaya
@oguzhan10
2.4.6
spark version 2.4.6
scala version 2.11.12
Leif Warner
@LeifW
There's no DataFrame type any longer in that version - that was just a type alias for Dataset[Row], anyways - http://spark.apache.org/docs/2.4.6/api/scala/index.html#org.apache.spark.sql.Dataset
Oğuzhan Kaya
@oguzhan10
df3.writeStream.outputMode("update").foreachBatch{ (batchDF:DataFrame, batchId:Long) =>
    val batchDF_1 = batchDF.withColumn("batch_id", lit(batchId))

    batchDF_1.write.format("mongo").mode("append").option("uri","mongodb://IP").option("database","local").option("collection","spark").save()}.start()
you mean this works
df3.writeStream.outputMode("update").foreachBatch{ (batchDF:Dataset[Row], batchId:Long) =>    
    val batchDF_1 = batchDF.withColumn("batch_id", lit(batchId))    
    batchDF_1.write.format("mongo").mode("append").option("uri","mongodb://IP").option("database","local").option("collection","spark").save()}.start()
error: not found: type Dataset
df3.writeStream.outputMode("update").foreachBatch{ (batchDF:Dataset[Row], batchId:Long) =>
Leif Warner
@LeifW
Yeah, should. Do you even need those type ascriptions on those lambda params? Would just foreachBatch (batchDF, batchId) => ... work
import org.apache.spark.sql.{Dataset, Row}
Oğuzhan Kaya
@oguzhan10
thanks mate
Oğuzhan Kaya
@oguzhan10
but now I am getting this error Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=
Seth Tisue
@SethTisue

Also, it's a shame sbt dependency graph doesn't work on sbt 1.3

@LeifW 0.10.0-RC1 doesn't work for you? https://github.com/jrudolph/sbt-dependency-graph/issues/178#issuecomment-645652112

Luis Miguel Mejía Suárez
@BalmungSan
Uhm Scala 2.13.3 is still not in listed in sdkman.
@SethTisue what would be the correct place to report that kind of things?
Seth Tisue
@SethTisue
@BalmungSan I have no clue how stuff ends up in sdkman. it's never come up before. I guess some volunteer has been doing it on their own initiative?
this seems to be how 2.13.2 got in: sdkman/sdkman-db-migrations#257
Luis Miguel Mejía Suárez
@BalmungSan
@SethTisue uhm, curious, for what I understand sdkman let vendors publish their new versions, and since the official Scala download page mentions sdkman I thought it was part of the release process.
Seth Tisue
@SethTisue
We aren't responsible for many of the other methods on that download page list either (homebrew, macports, ammonite, ...). If one developed a problem with not being promptly updated with new versions and it were called to our attention, we'd probably remove it.
there was some debate about whether to list sdkman. see scala/scala-lang#957 and scala/scala-lang#966
Luis Miguel Mejía Suárez
@BalmungSan
Makes sense, sorry for the assumption.
Luis Miguel Mejía Suárez
@BalmungSan

In any case, I opened the PR for the inclusion of 2.13.3: sdkman/sdkman-db-migrations#285

I was searching in the wrong repo how to do that, so thanks for pointing out the 2.13.2 one Seth :slight_smile:

Siddhant Sanyam
@siddhant3s
Is Zinc incremental compiler basically what SBT's incremental compilation as described here is but in a separate package? Do they follow similar heuristics?
Does Bloop use Zinc?
What is the difference between compilation speed of say someone long-running SBT vs using Bloop?
Henry Story
@bblfish
Hi, Is there a good library to help write SQL statements to a file. I am really just trying to make sure I escape strings correctly. I want to do this without having to open a JDBC connection.
Rob Norris
@tpolecat
You want to verify that a string contains well-formed SQL?
Henry Story
@bblfish
yes, at its simplest I want to make sure the strings are well quoted
I have produced output like this:
INSERT INTO OneTable VALUES
( '<https://www.omg.org/spec/LCC/Countries/Regions/ISO3166-2-SubdivisionCodes-GB/EileanSiar>',
  "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>",
  '<http://www.w3.org/2002/07/owl#NamedIndividual>'
),
( '<https://www.omg.org/spec/LCC/Countries/Regions/ISO3166-2-SubdivisionCodes-GB/EileanSiar>',
  "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>",
  '<https://www.omg.org/spec/LCC/Countries/CountryRepresentation/CountrySubdivision>'
),
( '<https://www.omg.org/spec/LCC/Countries/ISO3166-1-CountryCodes/Bulgaria>',
  "<https://www.omg.org/spec/LCC/Countries/CountryRepresentation/isIndependent>",
  '"true"^^<http://www.w3.org/2001/XMLSchema#boolean>'
), ...
but mysql won't accept that as it is not well quoted.
The above in an INSERT into the following table @tpolecat
CREATE TABLE OneTable(
   subject MEDIUMTEXT,
   relation MEDIUMTEXT,
   object MEDIUMTEXT
);
jdbc should be the standard way to quote strings, but it requires a db connection object. And I just want to produce a sql file.
Mario Galic
@mario-galic
@siddhant3s Zinc was once inbuilt just into sbt, and then they factored it out into a separate dependency which other projects can reuse, so the heuristics described in those docs should still hold. Bloop does use Zinc, you can see it being a dependency of bloop here. An aim of bloop is to improve upon zinc and be faster than compiling even within sbt interactive shell. Bloop focuses on compilation and testing cycles, as opposed the full range of build tasks, such as for example dependency resolution etc. You export the existing build project structure to bloop, and from then on one can use bloop for incremental compilation whilst still using the build tool for other build tasks if necessary.
Siddhant Sanyam
@siddhant3s
@mario-galic Wow thanks Mario for the in-depth answer to my questions. Is there a way for SBT to re-use bloop's compilation results so that when I do sbt run, I don't have to re-compile those files which Bloop already did for me.
Mario Galic
@mario-galic
Why not just run it with bloop? Something like bloop run --main example.Main myProject
Luis Miguel Mejía Suárez
@BalmungSan

@siddhant3s

Is there a way for SBT to re-use bloop's compilation results

AFAIK not yet, but it was work in progress.

But as Mario said, why not just using the bloop cli?
I just use SBT in local for two things, cross testing and assembly.
Siddhant Sanyam
@siddhant3s
I see. Well, currently bloop has been working transparently for me in the background. Metals makes it super simple that I don't even realize that bloop is present. It would be nice if my users don't need bloop cli for building or learn another command. Not a big deal but just nice to have. Also, if SBT could re-use Bloop's compilation then the assembly would be much faster.
I guess I can (or the bloop plugin for SBT can) probably add commands that can proxy bloop run.
@BalmungSan How do you run your project ? Do you use bloop cli or do you use the VSCode's Run functionality. The problem with VSCode's/Metal's Run functionality is that it doesn't run it in the terminal so you can't interact with the console so nicely (as I discovered)
Rob Norris
@tpolecat
I use vs-code and run/test inside metals most of the time, and run bloop/sbt in a console pane if I need to.
Siddhant Sanyam
@siddhant3s
Yeah. I think I was doing the same until I found out that there is no way to take console input from VSCode run. I wish that wasn't the case. Not that my current use case require Console input but it's just that I want something that I can use consistently regardless. I think I'm going to go with creating some SBT commands for my users. That might be the easiest way. Apparently Bloops folks are working on it.
Luis Miguel Mejía Suárez
@BalmungSan
@siddhant3s sorry for taking too long. I run test from terminal using either bloop or sbt.
Bloop when I do not cross-build and SBT when I do.
And for executing the application, I have only worked on http apis so I assembly then and then execute them with an script.
I agree it would be ideal if SBT could connect to bloop but as far as I understand it, it was quite complex.
I am looking forward for the implementation of BSP directly in SBT and see how it works with metals and VSCode.
If it works as good as with bloop, I probably will end up falling back to it and removing bloop from my system.
Siddhant Sanyam
@siddhant3s

I am looking forward for the implementation of BSP directly in SBT and see how it works with metals and VSCode.

That's very interesting. Mill apparently can act as a BSP server. Wonder if folks have tried using Metals with Mill has the build server.

Chris Coppins
@st900_gitlab
hi, is there an easiest / best way of using sealed classes e.g. below does not work due to Range being sealed? I've looked at a few methods but all seem a bit verbose
class SpecialRange(size: Int) extends Range(-size, size, 1) { require (size >= 0) }
Soren
@srnb_gitlab

Why not

object SpecialRange {

  def apply(size: Int): Range[Int] = (-size to size)

}

?
@st900_gitlab

Chris Coppins
@st900_gitlab
i was hoping to do something like object Five = SpecialRange(5) and then for (i <- Five) .... This would work I think but would have to define the iterator too - so I thought inheritance might save defining an iterator by delegation.