Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Arthur Stemmer
    Is there a list containing past occurrences of security vulnerabilities (CVEs) publicly available?
    Rahul Goyal
    does Toree support magic commands similar to sparkmagic: %config which will let notebook users specify spark configs dynamically?
    Kevin Bates
    Since the toree kernel’s startup creates the spark context (using parameters conveyed via the kernel spec and available via sc) I think it would be too late to apply magics for this. It looks like toree’s support for magics are more at the line and cell level with the notion that the context is already established. Copying @lresende for confirmation/correction.
    Rahul Goyal
    I agree with you @kevin-bates that the toree kernel would already have started the spark context and we can not apply config level magics...based on my limited understanding of "sparkmagic kernel", this problem is solved by delaying the creation of spark context until needed..that give scope to execute magic cells with "%config" and the kernel accumulates it.. it is not possible to do that same with Toree?
    Kevin Bates
    I don’t think so - but need @lresende to confirm.
    Luciano Resende
    if you set toree options to not start the context, you can create your own with SparkSession
    Kevin Bates
    Rahul Goyal
    how do i do that @lresende .. is there any link for reference?
    Luciano Resende
    Note that I just ssaw a comment there which implies that there might be a bug on the code … would appreciate help investigating and providing a patch
    David M.
    I have installed Apache Toree on Jupyter:
    pip install toree
    jupyter toree install --spark_home=/usr/local/spark-3.2.0-bin-hadoop3.2/
    but kernel crashes after start:
    Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
    at org.apache.toree.Main$.<init>(Main.scala:24)
    at org.apache.toree.Main$.<clinit>(Main.scala)
    at org.apache.toree.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: scala.App$class
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 15 more

    Kindly ask to advise how to fix it.

    Hi @s3uzz, my guess would be that you are using perhaps a version like 0.4 ? Which is still on Scala 2.11 and if you have hadoop 3.2.x you have Scala 2.12.

    Compile a newer version for Scala 2.12 support.

    David M.
    Thank you. My fault, version from pypi.org is 0.4.0 from Aug 2020. Just followed documentation...
    5 replies
    David M.

    Hi! Another problem using 0.5.0-rc4 with Spark 3.2.0. Please advise how to resolve.

    Exception in thread "main" scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.
    at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:24)
    at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:25)
    at scala.reflect.internal.Mirrors$RootsBase.$anonfun$getModuleOrClass$5(Mirrors.scala:61)
    at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:61)
    at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:198)
    at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:198)
    at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:199)
    at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:199)
    at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1251)
    at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1250)
    at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1408)
    at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1407)
    at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1450)
    at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1450)
    at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1506)
    at scala.tools.nsc.Global$Run.<init>(Global.scala:1213)
    at scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:432)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:814)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:772)
    at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:637)
    at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.$anonfun$start$1(ScalaInterpreterSpecific.scala:291)
    at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
    at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.start(ScalaInterpreterSpecific.scala:282)
    at org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific.start$(ScalaInterpreterSpecific.scala:266)
    at org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.start(ScalaInterpreter.scala:43)
    at org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.init(ScalaInterpreter.scala:94)
    at org.apache.toree.boot.layer.InterpreterManager.$anonfun$initializeInterpreters$1(InterpreterManager.scala:35)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
    at org.apache.toree.boot.layer.InterpreterManager.initializeInterpreters(InterpreterManager.scala:34)
    at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents(ComponentInitialization.scala:87)
    at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents$(ComponentInitialization.scala:69)
    at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
    at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:102)
    at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:35)
    at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
    at scala.Function0.apply$mcV$sp(Function0.scala:39)
    at scala.Function0.apply$mcV$sp$(Function0.scala:39)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
    at scala.App.$anonfun$main$1$adapted(App.scala:80)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at scala.App.main(App.scala:80)
    at scala.App.main$(App.scala:78)
    at org.apache.toree.Main$.main(Mai

    Should I downgrade Java version? I use openjdk 11.0.13 2021-10-19
    WARN Main$$anon$1: No external magics provided to PluginManager!
    [init] error: error while loading Object, Missing dependency 'class scala.native in compiler mirror', required by /modules/java.base/java/lang/Object.class
    David M.
    ok, I have to downgrade Java 11 -> 8
    6 replies
    David M.
    I install Toree with 'jupyter toree install --spark_home=${SPARK_HOME} --interpreters=Scala,PySpark,SQL --python_exec=/opt/conda/bin/python3.8'
    However, in Jupyterlab I see only two kernels: Scala and SQL.
    Kindly ask how to debug. Thanks in advance.
    Toree version 0.4.0
    David M.
    It seems that I have to downgrade Python 3.8->3.7
    David M.
    [ToreeInstall] ERROR | Unknown interpreter PySpark. Skipping installation of PySpark interpreter

    Hi @s3uzz

    Yes you could try an older python, I am still using 3.6 as mentioned previously.

    Below is a small example bash script I use to install

    export VERSION=0.1.0
    export SPARK_HOME=/path/to/sparkHome
    jupyter-toree install \
        --replace \
        --debug \
        --user \
        --kernel_name "project $VERSION" \
        --spark_home=${SPARK_HOME} \
        --spark_opts="--master yarn --jars $jars"
    David M.
    Zakk, thank you. Maybe intepreter PySpark is not available in version 0.4.0> I use
    jupyter toree install --spark_home=${SPARK_HOME} --interpreters=Scala,PySpark,SQL --python_exec=/opt/conda/bin/python3.8
    and get error
    [ToreeInstall] ERROR | Unknown interpreter PySpark. Skipping installation of PySpark interpreter
    5 replies
    Hi, I am running toree 0.5.0 rc4 with Spark 3.1.2, everything is working fine execept there is a problem with errors not displaying in the cell output while they do show up in the text if I do "Download as notebook .pynb". For example if I type "blah" in the cell and run it I see a blank response in the notebook, but this value in the downloaded file - {
    "cell_type": "code",
    "execution_count": 2,
    "id": "68eb215f",
    "metadata": {},
    "outputs": [
    "ename": "Compile Error",
    "evalue": "<console>:26: error: not found: value blah\n blah\n ^\n",
    "output_type": "error",
    "traceback": []
    "source": [
    } Is this a bug or am I missing some configuration? How do I get these types of error to display in the cell output?
    Rahul Goyal
    Hi team
    the issue that @amitzo has brought up, we are seeing similar issue and i think this is preventing a the end users from getting to know what exactly happened in the backend..
    is there a ticket open already on this ? what is the plan in general to improve and address user experience gaps from the community.
    I will be more than happy to help out if someone can guide on this.
    Kevin Bates

    Here’s a link to a related Toree JIRA: https://issues.apache.org/jira/browse/TOREE-522?jql=project%20%3D%20TOREE%20AND%20component%20%3D%20Kernel
    In there @lresende points out that this appears to be a function of the front-end - whether it is Notebook or Lab. I see the same behavior. In fact, when I run using the classic Notebook front-end, I don’t see the error when its syntax related (e.g., “blah”), but do get some error result for something like a divide by zero (e.g., “1/0”). However, what is interesting is that if I save that notebook and then open it in Lab, I see the appropriate messages in both cases. So, as @amitzo is pointing out, the error details are persisted in the notebook, but something about the “display” that’s not working.
    I’m curious if you see any error output resulting from divide-by-zero issues?

    Here are two screen shots, the first using Notebook and the second using Lab where, for the second, I merely opened the notebook file produced from the first...

    Screen Shot 2022-05-03 at 11.36.19 AM.png
    Screen Shot 2022-05-03 at 11.38.28 AM.png
    Omkar Kalange
    Kernel dies and does not restart
    kernel.json : {
    "argv": [
    "env": {
    "TOREE_SPARK_OPTS": "--master=spark://ip-172-31-1-122.ec2.internal:7077",
    "TOREE_OPTS": "",
    "SPARK_HOME": "/home/ec2-user/spark"
    "display_name": "Apache Toree - Scala",
    "language": "scala",
    "interrupt_mode": "signal",
    "metadata": {}
    Luciano Resende
    well, information from the actuall logs or anything related to failures would be much more helpful
    Omkar Kalange
    AsyncIOLoopKernelRestarter: restart failed
    I tried uninstalling jupyter and toree
    installed it back but still same error
    Kevin Bates
    Hi @komkar123 - thanks for the additional information. The issue is that the Toree kernel instance can’t get fully started. As a result, the kernel restarter detects its death and attempts to start the kernel again (4 more times). So the Jupyter side of things is working as expected and you should focus on resolving the issue in the stack trace. Is there any additional information before or after what is pictured? I’m assuming this occurs immediately at startup, before any cell is executed, but please let us know if that’s not the case.
    Omkar Kalange
    Yes, this happens when I select Apache Toree as kernel in Jupyter, before executing any piece of code.
    Omkar Kalange


    This is what happens when I start jupyter and select Apache Toree as kernel

    Omkar Kalange

    jupyter toree install --user --spark_home=$HOME/spark --spark_opts="--master=spark://$MASTER_URL:7077"

    Is in stack to install toree

    Omkar Kalange
    I searched for the error : java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V and it looks like there is a version mismatch of scala
    Luciano Resende
    What version of Scala are you using? For latest Toree you should be using Scala 2.12
    Omkar Kalange
    Omkar Kalange
    Scala code runner version 2.12.2 -- Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.
    This is the version of scala installed on my machine. However, when I start spark 2.4.6 using spark-shell, it uses scala 2.11.12
    Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)
    Luciano Resende
    You should install Toree 0.4 for Spark 2.4.6 and Scala 2.11
    Toree 0.5, by default, is build with Spark 3 + Scala 2.12
    Omkar Kalange
    Oh, can u please tell me how can I run install toree command with 0.4?
    Luciano Resende
    pip install toree==0.4.0