deanwampler/JustEnoughScalaForSpark

Discuss Scala APIs and idioms for Spark Programming

Dean Wampler

@deanwampler

I don’t have any recommendations, other than the fact you can share code between batch and streaming. You might ask the https://gitter.im/spark-scala/Lobby channel

sud1489

@sud1489

I am trying to run wordFileNameOnes.count but it's throwing the exception...
Caused by: java.util.regex.PatternSyntaxException: Unexpected internal error near index 1
I am running this in spark-shell launched from windows.. is something wrong with the regex split("""\W+""") given in the code when running from windows?

Dean Wampler

@deanwampler

That shouldn’t happen even on Windows. Can you provide a stack trace? Did you make any modifications that you think might have caused the issue?

sud1489

@sud1489

hi Dean.. there was issue with the separator which was being used to get the file name from the full file path... it's resolved now and getting results as expected...

Fernando Margueirat

@adatasetaday

I see there has not been any new messages in almost a year, but I hope someone still checks this from time to time. I am running the Jupyter notebook as per https://github.com/deanwampler/JustEnoughScalaForSpark but on the cell's output it does not print the types of the variables and it is sometimes hard to follow the examples without that information.

i.e. instead of printing
(Int, Int) = (1,2)
it prints
(1,2)

Does anyone knows if this is some kind of setting I can change or it is due to me having a different version of the Docker container?

Fernando Margueirat

@adatasetaday

I double checked that I ran %showTypes

Fernando Margueirat

@adatasetaday

Never mind, apparently there's a bug https://issues.apache.org/jira/browse/TOREE-467

Rickyhai

@rickyhai11

Dear everyone,
I am trying to generate uber-jar using sbt complie and sbt package commands for running my application on our remote server with spark installed as standalone mode there. I used deeplearning4j framework for building LSTM neural network and tend to perform traing model through spark. Nevertheless, I got into issue when running spark-submit command:

spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-core:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"

The problem is that seemly spark-submit did not take effect in my circumstance. It has been done right after entering spark-submit withou throwing any error. I have seen the progress of training in the ouput.

[hadoop@gaion34 lstm_train]$ spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-core:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"
2018-04-25 17:06:50 WARN  Utils:66 - Your hostname, gaion34 resolves to a loopback address: 127.0.0.1; using 192.168.0.173 instead (on interface eno1)
2018-04-25 17:06:50 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-04-25 17:06:51 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-25 17:06:51 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-04-25 17:06:51 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-c4aee15e-d23b-4c03-95a7-12d9d39f714a
[hadoop@abc lstm_train]$ spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-nn:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"
2018-04-25 17:07:12 WARN  Utils:66 - Your hostname, abcresolves to a loopback address: 127.0.0.1; using 192.168.0.11 instead (on interface eno1)
2018-04-25 17:07:12 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-04-25 17:07:13 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-25 17:07:13 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-04-25 17:07:13 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-82fdaebf-1121-4e31-8c4f-37aea9683922

my main class:


object SparkLSTM {
  def main(args: Array[String])= {
    if (args.length == 2) {
      val filePath = args(0)    //"/Users/kym1992/STUDY/NEU/CSYE7200/Dataset/nyse/prices-split-adjusted.csv"
      val symbolName = args(1)
      BasicConfigurator.configure()
      val prepared = StockPricePredictionLSTM.prepare(filePath, symbolName, 0.90)
      val result = StockPricePredictionLSTM.predictPriceOneAhead(prepared._1, prepared._2, prepared._3, prepared._4, prepared._5)
      println("predicts, actual")
      (result.predicts, result.actuals).zipped.foreach((x, y) => println(x + ", " + y))
      saveAsCsv(result, symbolName)

      result.predicts.foreach(r => println(r))
    }
  }

Any one has experienced this issue before, please advise me . thanks

Rickyhai

@rickyhai11

I have not seen the progress of training in the ouput.

Dean Wampler

@deanwampler

Hi, @rickyhai11. I see you asked on the Spark with Scala channel. That's a better place for general questions not specific to the "Just Enough..." tutorial. Good luck!

abhisam

@abhisam

How does spark create external table works ?
Does it create external location with table ?
Please help me on this

Dean Wampler

@deanwampler

It should, but I haven't tried it in a while. Try the https://gitter.im/spark-scala/Lobby channel, if you have problems. It has a lot more people participating.

abhisam

@abhisam

@deanwampler would you know how to write Scala test cases for object variables ?

Dean Wampler

@deanwampler

They are like instance static class variables in Java, you just reference them with MyClass.variable, instead of myInstance.variable. With that, you can test them like any other variables.

abhisam

@abhisam

@deanwampler is it possible to override Scala object method for test cases ?

Dean Wampler

@deanwampler

No, they can't be overridden. You'll need to extract the method into a separate trait or class.

abhisam

@abhisam

@deanwampler you mean I have to change main code object to trait or class ?

Dean Wampler

@deanwampler

If you like the tutorial that goes with this Gitter channel, I just updated my other tutorial, which teaches Spark using Scala. It now has Jupyter notebook examples, like this tutorial has used for a while. https://github.com/deanwampler/spark-scala-tutorial

ksg

@ksg97031

Hi~

Dean Wampler

@deanwampler

I just merged in some improvements to how Jupyter is used and the instructions in the README.

Dean Wampler

@deanwampler

I added the option to use BeakerX instead of the Jupyter image. The README provides details. I also added an HTML printout of the tutorial notebook to go with the PDF printout. Otherwise, no changes to the tutorial itself.

Dean Wampler

@deanwampler

Well I was a bit premature with the BeakerX support. Had to revert that commit and rework it on a branch. Stay tuned...

Dean Wampler

@deanwampler

Scala 2.12 support was just merged into Spark master. https://twitter.com/deanwampler/status/1025030404078796800

MuralikrishnaDanam

@MuralikrishnaDanam

Hello i just started my docker container for jupyter UI i followed all instruction when i logged into UI . "work" folder is empty , could you please let me know what went wrong

MuralikrishnaDanam

@MuralikrishnaDanam

am sorry i figured it out

Dean Wampler

@deanwampler

Glad you figured it out.

François Sarradin

@fsarradin

Hi @deanwampler with my colleagues we have done a translation in French of your tutorial. It has been played yesterday for the first time. The tutorial just worked well and has met with approval. So we would like to thank for your great tutorial.
The translation is available here: https://github.com/univalence/CeQuilFautDeScalaPourSpark

Dean Wampler

@deanwampler

Awesome! Thanks for letting me know. I added a link in the README!

Czkonverse

@Czkonverse

Hello~It's very grateful to get the course. But I met a problem here, when I run the file run.bat, the outpu is "Unable to find image 'jupyter/all-spark-notebook:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'."

I'm running in a windows 10 64-bit system, does anyone meet the same problem?

@deanwampler

Dean Wampler

@deanwampler

I’m guessing your local docker environment couldn’t download the image from Docker hub. Are you behind a network proxy, by chance? Can you pull other docker images? Example: docker pull hello-world pulls a very small test image. If you suspect problems pulling, this documentation should help: https://docs.docker.com/engine/reference/commandline/pull/. Note the proxy configuration section.

Czkonverse

@Czkonverse

@deanwampler Thank U for your advice, I solved the problem after I configured the docker settings in Daemon, I have runed the jupyter notebook successfully, and thank U for the course again, thanks.

Dean Wampler

@deanwampler

Great! good luck.

mengd2

@mengd2

Hi @deanwampler . I found there is a token authentication now when trying to open the jupyter notebook. I also noticed that the docker image was updated 2days ago. Did someone enable the token authentication and then commit it?

Dean Wampler

@deanwampler

I don't have any input in how the Jupyter Docker images are built, etc., but it's standard practice now to require the token to open the GUI. I just tried the new image and it worked fine for me, but you have to use the full URL printed to your terminal, not just localhost:8888. In some terminal environments you can control/command click the URL to do this without having to copy and paste. Let me know if this doesn't work.

Dean Wampler

@deanwampler

I just pushed a minor update, mostly fixing typos and updating the text in a few places. Mostly, I confirmed the tutorial still works correctly with the latest Docker image, which uses Spark 2.4.5.

rajaniesh

@rajaniesh

Hi @deanwampler how to use these samples in databricks for azure scala notebook?

Dean Wampler

@deanwampler

I don't know. Does their environment support importing iPython notebooks? If not, you would have to copy and paste the cells.

Lucien

@lucien-paypay

Hello here, I'm running the Docker image and found that I can't run %showTypes on in third cell, the notebook gives me this error, could anyone help? Thank you!

<console>:26: error: not found: value %
       %showTypes on
       ^
<console>:26: error: not found: value on
       %showTypes on
                  ^

Dean Wampler

@deanwampler

It appears that none of "magics" like this work anymore :( I just pushed an update to the notebook with comments about this and workarounds. For example, %showTypes on isn't really needed; the types are shown anyway. So, I removed that cell. Later on, you'll run into problems with %%dataframe, but I added cells showing the alternative way to render a DataFrame, using the show() method.

Thanks for pointing out this issue.

Lucien

@lucien-paypay

@deanwampler Thank you. I thought it was because different Spark version, and trying to build jupyter/all-spark-notebook with Spark 2.4. By the way, the current instruction will lead the reader to Spark 3.0.1, not sure if there will be any conflict with the notebook content. Anyway, thank you for sharing this notebook, it's helpful!

Dean Wampler

@deanwampler

I ran the whole notebook and all the Spark cells still work. However, the content is probably a little old.

Lucien

@lucien-paypay

Cool, thank you!

Dean Wampler

@deanwampler

<shameless-plug>I'm pleased to announce that Programming Scala, Third Edition (O'Reilly) is now published! Ebooks will be available within the next week and print copies a few weeks from now. You can find details at http://programming-scala.com, including links to the code examples on github.</shameless-plug>

Where communities thrive

People

Repo info

Activity