Does anyone knows if this is some kind of setting I can change or it is due to me having a different version of the Docker container?
Dear everyone,
I am trying to generate uber-jar using sbt complie and sbt package commands for running my application on our remote server with spark installed as standalone mode there. I used deeplearning4j framework for building LSTM neural network and tend to perform traing model through spark. Nevertheless, I got into issue when running spark-submit command:
spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-core:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"The problem is that seemly spark-submit did not take effect in my circumstance. It has been done right after entering spark-submit withou throwing any error. I have seen the progress of training in the ouput.
[hadoop@gaion34 lstm_train]$ spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-core:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"
2018-04-25 17:06:50 WARN Utils:66 - Your hostname, gaion34 resolves to a loopback address: 127.0.0.1; using 192.168.0.173 instead (on interface eno1)
2018-04-25 17:06:50 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-04-25 17:06:51 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-25 17:06:51 INFO ShutdownHookManager:54 - Shutdown hook called
2018-04-25 17:06:51 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-c4aee15e-d23b-4c03-95a7-12d9d39f714a
[hadoop@abc lstm_train]$ spark-submit --class "lstm.SparkLSTM" --master local[*] stock_prediction_scala_2.11-0.1.jar --packages org.deeplearning4j:deeplearning4j-nn:0.9.1 "/home/hadoop/ScalaWorkspace/Stock_Prediction_Scala/target/lstm_train/prices-split-adjusted.csv" "WLTW"
2018-04-25 17:07:12 WARN Utils:66 - Your hostname, abcresolves to a loopback address: 127.0.0.1; using 192.168.0.11 instead (on interface eno1)
2018-04-25 17:07:12 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-04-25 17:07:13 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-25 17:07:13 INFO ShutdownHookManager:54 - Shutdown hook called
2018-04-25 17:07:13 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-82fdaebf-1121-4e31-8c4f-37aea9683922my main class:
object SparkLSTM {
def main(args: Array[String])= {
if (args.length == 2) {
val filePath = args(0) //"/Users/kym1992/STUDY/NEU/CSYE7200/Dataset/nyse/prices-split-adjusted.csv"
val symbolName = args(1)
BasicConfigurator.configure()
val prepared = StockPricePredictionLSTM.prepare(filePath, symbolName, 0.90)
val result = StockPricePredictionLSTM.predictPriceOneAhead(prepared._1, prepared._2, prepared._3, prepared._4, prepared._5)
println("predicts, actual")
(result.predicts, result.actuals).zipped.foreach((x, y) => println(x + ", " + y))
saveAsCsv(result, symbolName)
result.predicts.foreach(r => println(r))
}
}Any one has experienced this issue before, please advise me . thanks
It should, but I haven't tried it in a while. Try the https://gitter.im/spark-scala/Lobby channel, if you have problems. It has a lot more people participating.
@deanwampler would you know how to write Scala test cases for object variables ?
If you like the tutorial that goes with this Gitter channel, I just updated my other tutorial, which teaches Spark using Scala. It now has Jupyter notebook examples, like this tutorial has used for a while. https://github.com/deanwampler/spark-scala-tutorial
I added the option to use BeakerX instead of the Jupyter image. The README provides details. I also added an HTML printout of the tutorial notebook to go with the PDF printout. Otherwise, no changes to the tutorial itself.
Scala 2.12 support was just merged into Spark master. https://twitter.com/deanwampler/status/1025030404078796800
Hello i just started my docker container for jupyter UI i followed all instruction when i logged into UI . "work" folder is empty , could you please let me know what went wrong
Hi @deanwampler with my colleagues we have done a translation in French of your tutorial. It has been played yesterday for the first time. The tutorial just worked well and has met with approval. So we would like to thank for your great tutorial.
The translation is available here: https://github.com/univalence/CeQuilFautDeScalaPourSpark
The translation is available here: https://github.com/univalence/CeQuilFautDeScalaPourSpark
Hello~It's very grateful to get the course. But I met a problem here, when I run the file run.bat, the outpu is "Unable to find image 'jupyter/all-spark-notebook:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'."
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'."
I'm running in a windows 10 64-bit system, does anyone meet the same problem?
@deanwampler
I’m guessing your local docker environment couldn’t download the image from Docker hub. Are you behind a network proxy, by chance? Can you pull other docker images? Example:
docker pull hello-world pulls a very small test image. If you suspect problems pulling, this documentation should help: https://docs.docker.com/engine/reference/commandline/pull/. Note the proxy configuration section.
I don't have any input in how the Jupyter Docker images are built, etc., but it's standard practice now to require the token to open the GUI. I just tried the new image and it worked fine for me, but you have to use the full URL printed to your terminal, not just
localhost:8888. In some terminal environments you can control/command click the URL to do this without having to copy and paste. Let me know if this doesn't work.
It appears that none of "magics" like this work anymore :( I just pushed an update to the notebook with comments about this and workarounds. For example,
%showTypes on isn't really needed; the types are shown anyway. So, I removed that cell. Later on, you'll run into problems with %%dataframe, but I added cells showing the alternative way to render a DataFrame, using the show() method.
Thanks for pointing out this issue.
@deanwampler Thank you. I thought it was because different Spark version, and trying to build
jupyter/all-spark-notebook with Spark 2.4. By the way, the current instruction will lead the reader to Spark 3.0.1, not sure if there will be any conflict with the notebook content. Anyway, thank you for sharing this notebook, it's helpful!
<shameless-plug>I'm pleased to announce that Programming Scala, Third Edition (O'Reilly) is now published! Ebooks will be available within the next week and print copies a few weeks from now. You can find details at http://programming-scala.com, including links to the code examples on github.</shameless-plug>
Hi Guys
We have open-sourced a small library - spark-property-tests for writing easy tests on spark dataframes. We have been using it internally at zeotap data engineering for some time now and thought that the community might benefit from it.
We have published the artifacts in maven central for spark 2.4.x and other versions are on the way. Please review the utility and let us know your feedback.
I have problem, the JustEnoughScalaForSpark.ipynb doesn't run with Scala
only with Python 3(ipykernel)