A kernel that enables applications to interact with Apache Spark (http://toree.apache.org/)
https://issues.apache.org/jira/projects/TOREE/issues/TOREE-522
Any workarounds?
Hello. I am trying to use toree 0.4.0 with jupyterhub 1.1.0. Everything looks great, but i can't see any exceptions traces from the code that i'm running in nb. What can i try, how can i debug this issue?
I am facing similar issue. I can see few lines out of whole stack trace. I am using toree 0.6.0.
sc) I think it would be too late to apply magics for this. It looks like toree’s support for magics are more at the line and cell level with the notion that the context is already established. Copying @lresende for confirmation/correction.
I have installed Apache Toree on Jupyter:
pip install toree
jupyter toree install --spark_home=/usr/local/spark-3.2.0-bin-hadoop3.2/
but kernel crashes after start:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
at org.apache.toree.Main$.<init>(Main.scala:24)
at org.apache.toree.Main$.<clinit>(Main.scala)
at org.apache.toree.Main.main(Main.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.App$class
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 15 more
Kindly ask to advise how to fix it.
Hi! Another problem using 0.5.0-rc4 with Spark 3.2.0. Please advise how to resolve.
Exception in thread "main" scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:24)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:25)
at scala.reflect.internal.Mirrors$RootsBase.$anonfun$getModuleOrClass$5(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:61)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:198)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:198)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:199)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:199)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1251)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1250)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1408)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1407)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1450)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1450)
at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1506)
at scala.tools.nsc.Global$Run.<init>(Global.scala:1213)
at scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:432)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:814)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:772)
at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:637)
at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.$anonfun$start$1(ScalaInterpreterSpecific.scala:291)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.start(ScalaInterpreterSpecific.scala:282)
at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.start$(ScalaInterpreterSpecific.scala:266)
at org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.start(ScalaInterpreter.scala:43)
at org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.init(ScalaInterpreter.scala:94)
at org.apache.toree.boot.layer.InterpreterManager.$anonfun$initializeInterpreters$1(InterpreterManager.scala:35)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
at org.apache.toree.boot.layer.InterpreterManager.initializeInterpreters(InterpreterManager.scala:34)
at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents(ComponentInitialization.scala:87)
at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents$(ComponentInitialization.scala:69)
at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:102)
at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:35)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at org.apache.toree.Main$.main(Mai
[init] error: error while loading Object, Missing dependency 'class scala.native in compiler mirror', required by /modules/java.base/java/lang/Object.class
Hi @s3uzz
Yes you could try an older python, I am still using 3.6 as mentioned previously.
Below is a small example bash script I use to install
#!/bin/sh
export VERSION=0.1.0
export SPARK_HOME=/path/to/sparkHome
jars="file:///path/to/jar"
jars="$jars,file:///path/to/jar"
jupyter-toree install \
--replace \
--debug \
--user \
--kernel_name "project $VERSION" \
--spark_home=${SPARK_HOME} \
--spark_opts="--master yarn --jars $jars"jupyter toree install --spark_home=${SPARK_HOME} --interpreters=Scala,PySpark,SQL --python_exec=/opt/conda/bin/python3.8
and get error
[ToreeInstall] ERROR | Unknown interpreter PySpark. Skipping installation of PySpark interpreter
5 replies
"cell_type": "code",
"execution_count": 2,
"id": "68eb215f",
"metadata": {},
"outputs": [
{
"ename": "Compile Error",
"evalue": "<console>:26: error: not found: value blah\n blah\n ^\n",
"output_type": "error",
"traceback": []
}
],
"source": [
"blah"
]
} Is this a bug or am I missing some configuration? How do I get these types of error to display in the cell output?
the issue that @amitzo has brought up, we are seeing similar issue and i think this is preventing a the end users from getting to know what exactly happened in the backend..
is there a ticket open already on this ? what is the plan in general to improve and address user experience gaps from the community.
I will be more than happy to help out if someone can guide on this.
thanks
Here’s a link to a related Toree JIRA: https://issues.apache.org/jira/browse/TOREE-522?jql=project%20%3D%20TOREE%20AND%20component%20%3D%20Kernel
In there @lresende points out that this appears to be a function of the front-end - whether it is Notebook or Lab. I see the same behavior. In fact, when I run using the classic Notebook front-end, I don’t see the error when its syntax related (e.g., “blah”), but do get some error result for something like a divide by zero (e.g., “1/0”). However, what is interesting is that if I save that notebook and then open it in Lab, I see the appropriate messages in both cases. So, as @amitzo is pointing out, the error details are persisted in the notebook, but something about the “display” that’s not working.
I’m curious if you see any error output resulting from divide-by-zero issues?
Here are two screen shots, the first using Notebook and the second using Lab where, for the second, I merely opened the notebook file produced from the first...
"argv": [
"/home/ec2-user/.local/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
"--profile",
"{connection_file}"
],
"env": {
"DEFAULT_INTERPRETER": "Scala",
"TOREE_SPARK_OPTS": "--master=spark://ip-172-31-1-122.ec2.internal:7077",
"TOREE_OPTS": "",
"SPARK_HOME": "/home/ec2-user/spark"
},
"display_name": "Apache Toree - Scala",
"language": "scala",
"interrupt_mode": "signal",
"metadata": {}
