Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    krajendrannv
    @krajendrannv
    thanks!
    metanil
    @metanil
    pretty new to Toree, How do I load jar in driver added by %AddJar ?
    bharathwgl
    @bharathwgl

    We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws Keberos error, we have a valid kerberos ticket in place. Here is the kernel.json

    {
    "argv": [
    "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
    "--profile",
    "{connection_file}"
    ],
    "env": {
    "DEFAULT_INTERPRETER": "Scala",
    "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster",
    "TOREE_OPTS": "",
    "HADOOP_CONF_DIR": "/etc/hadoop/conf",
    "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2",
    "PYTHONPATH": "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib",
    "PYTHON_EXEC": "python",
    "HBASE_CONF_DIR": "/etc/hbase/conf",
    "HBASE_HOME": "/usr/hdp/current/hbase-client",
    "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64"
    },
    "display_name": "Toree - Scala-kernel",
    "language": "scala",
    "interrupt_mode": "signal",
    "metadata": {}
    }

    Error Message:
    20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8, retries=36, started=18874 ms ago, cancelled=false, msg=Call to compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=compute-4.datalake.ntt

    With above configuration, kernel dies sometimes.

    Please suggest if the way we are passing configs in Kernel.Json is the right way ?

    Luciano Resende
    @lresende
    Could you please create an issue on the toree jira?
    Hadrien Kohl
    @hadrienk
    Hi. I am trying to use the toree jupyter kernel with cluster mode on a YARN cluster. I have found contradicting information on whether it is supported or not?
    In cluster mode, the driver program (kernel) will run on one of the YARN executor, it is not clear however how the communication between Jupyter and the kernel is established after the driver/kernel program starts?
    Kevin Bates
    @kevin-bates

    Hi @hadrienk. The Apache Toree kernel can be run in cluster mode via Jupyter Enterprise Gateway (EG). EG "packages" its launched kernels (Python, R, Scala) with a sibling (aka launcher) that communicates the ZeroMQ ports back to the EG server. These siblings also handle interrupt and shutdown operations, the latter in case the message-based shutdown doesn't succeed.

    EG essentially abstracts the kernel process used in notebook server such that resource-manager-specific management can take place. This abstraction is called a process-proxy and EG provides process-proxies for Hadoop YARN, Kubernetes, Docker Swarm, etc. In container-based environments, the kernels reside in their own images, thereby gaining much greater isolation.

    Shu SHANG
    @suesunss

    Hi, I am new to Apache Toree.

    I've installed Spark Scala Kernel. I noticed that for cells containing only comments, for instance:

    // this is a comment

    By running the cell, Toree will give an exception:

    Name: Syntax Error.
    Message: 
    StackTrace:

    This exception will stop the execution of the notebook if I use cron to periodically run the notebook.
    Is there a workaround for this ?

    Luciano Resende
    @lresende
    Mark the cell as Markdown, it will give you more flexibility
    V4G4X
    @V4G4X

    When I open jupyter and when it tries to start the kernel, it never manages to.
    And this is the exception I see occurring in the terminal used to open Jupyter.

    Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class at org.apache.toree.Main$.<init>(Main.scala:24) at org.apache.toree.Main$.<clinit>(Main.scala) at org.apache.toree.Main.main(Main.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: scala.App$class at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 15 more log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [W 13:58:54.636 LabApp] KernelRestarter: restart failed [W 13:58:54.637 LabApp] Kernel 79c546a8-9f84-457f-b0ba-9c370f6b2a0a died, removing from map.

    My Spark Version is 3.0.0 and Scala is 2.12.10

    Luciano Resende
    @lresende
    you need to build master using scala 2.12 to run on spark 3.0
    i am wrapping up a release to support that
    mainly some docker cleanup issues remaining
    V4G4X
    @V4G4X
    Thank you for replying
    Thank you so much for your work
    I will try that and reply back
    V4G4X
    @V4G4X

    make release gave this error below. So did sudo make jupyter during its final stages.

    APACHE_SPARK_VERSION=3.0.0 VERSION=0.5.0.dev1-incubating IS_SNAPSHOT=true sbt root/assembly  
    Unrecognized VM option 'CMSPermGenSweepingEnabled'  
    Error: Could not create the Java Virtual Machine.  
    Error: A fatal exception has occurred. Program will exit.  
    copying runtime jar...  
    mkdir: cannot create directory ‘’: No such file or directory  
    Unrecognized VM option 'CMSPermGenSweepingEnabled'  
    Error: Could not create the Java Virtual Machine.  
    Error: A fatal exception has occurred. Program will exit.  
    Unrecognized VM option 'CMSPermGenSweepingEnabled'  
    Error: Could not create the Java Virtual Machine.   
    Error: A fatal exception has occurred. Program will exit.   
    make: *** [Makefile:117: target/scala-2.12/toree-assembly-0.5.0.dev1-incubating-SNAPSHOT.jar] Error 1

    Having looked up Unrecognized VM option 'CMSPermGenSweepingEnabled'I saw that it is an issue people face when their sbt is not installed correctly.
    My sbt(1.3.13) is installed, I have used it.
    I also have Java(AdoptOpenJDK 11.0.8) and Scala(2.12.10) installed via SDKMAN!, they also run without a hitch.
    I freshly installed docker, it ran sudo docker run hello-world and sudo docker run -it ubuntu bash.
    JAVA_HOME, SCALA_HOME, and SPARK_HOME are all defined too.

    I am new to building from source and to docker.
    Any help would be greatly appreciated.

    V4G4X
    @V4G4X

    So I found this link: https://gitter.im/ibm-et/spark-kernel/archives/2016/11/22
    From back when Toree was spark-kernel.
    He gets my exact issue down the line.
    He was told to remove -XX:+CMSPermGenSweepingEnabled from .jvmopts.
    I did the same, and it moved forward.
    Yay!
    BUT
    This error came by:

    [info] Compiling 12 Scala sources and 6 Java sources to /home/v4g4x/incubator-toree/plugins/target/scala-2.12/classes ...
    [error] /home/v4g4x/incubator-toree/plugins/src/main/scala/org/apache/toree/plugins/PluginManager.scala:132:35: method newInstance in class Class is deprecated
    [error]       val tryInstance = Try(klass.newInstance())
    [error]                                   ^
    [error] one error found

    This seems to be an issue in the source code I believe.
    I don't know how to fix this, I will keep looking into it,
    And wait for your inputs at the same time.
    Thank You.

    https://stackoverflow.com/questions/46393863/what-to-use-instead-of-class-newinstance
    So this post showed a similar issue for Java, it is probably applicable for Scala too.
    So maybe I can go down to that specific Scala file, and modify that part.
    Except, is that recommended? Should I do that?
    V4G4X
    @V4G4X
    Meh, I'm trying it, let's see what happens.
    V4G4X
    @V4G4X
    Okay, so in the above file PluginManager.scala:
    I replaced val tryInstance = Try(klass.newInstance()) with val tryInstance = Try(klass.getDeclaredConstructor().newInstance()).
    I ran make release again and it compiled it this time.
    But there's a new error.
    I'll post details again.
    V4G4X
    @V4G4X
    [error] /home/v4g4x/incubator-toree/kernel-api/src/main/scala/org/apache/toree/dependencies/CoursierDependencyDownloader.scala:266:9: constructor Long in class Long is deprecated  
    [error]         new java.lang.Long(downloaded),  
    [error]         ^  
    [error] /home/v4g4x/incubator-toree/kernel-api/src/main/scala/org/apache/toree/dependencies/CoursierDependencyDownloader.scala:267:9: constructor Double in class Double is deprecated  
    [error]         new java.lang.Double(percent)  
    [error]         ^  
    [error] two errors found
    I'm gonna try using <Wrapper>.valueOf()instead of the <Wrapper>()constructor.

    Replaced

    new java.lang.Long(downloaded),   
    new java.lang.Double(percent)

    with

    java.lang.Long.valueOf(downloaded),  
    java.lang.Double.valueOf(percent)
    That worked
    Another Depracation Error:
    [info] Compiling 64 Scala sources to /home/v4g4x/incubator-toree/kernel/target/scala-2.12/classes ...  
    [error] /home/v4g4x/incubator-toree/kernel/src/main/scala/org/apache/toree/boot/layer/InterpreterManager.scala:102:12: method newInstance in class Class is deprecated  
    [error]           .newInstance().asInstanceOf[Interpreter]  
    [error]            ^  
    [error] one error found
    V4G4X
    @V4G4X
    We've encountered this, I'm fixing that up.
    Replacing
    .newInstance().asInstanceOf[Interpreter]
    with
    .getDeclaredConstructor().newInstance().asInstanceOf[Interpreter]
    Let's see now
    YESS!! Finally some success
    [success] Total time: 41 s, completed 20-Aug-2020, 11:18:12 AM 
    docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run  /docker.sock: connect: permission denied.  
    See 'docker run --help'.  
    make: *** [Makefile:234: dist/toree-pip/toree-0.5.0.dev1.tar.gz] Error 126
    V4G4X
    @V4G4X
    I will look into that Docker error.
    I have, so far, run Docker only with sudo.
    I will try running make release with sudo too.

    Okay so, sudo make release ran with a good amount of SUCCESS messages.
    Here's where that yucked up too.

    GPG is gpg and file is dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    gpg: directory '/root/.gnupg' created
    gpg: keybox '/root/.gnupg/pubring.kbx' created
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key
    GPG is gpg and file is dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key
    GPG is gpg and file is dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key
    GPG is gpg and file is dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key
    GPG is gpg and file is dist/toree-pip/toree-0.5.0.dev1.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-pip/toree-0.5.0.dev1.tar.gz
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key
    GPG is gpg and file is dist/toree-pip/toree-0.5.0.dev1.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-pip/toree-0.5.0.dev1.tar.gz
    gpg: no default secret key: No secret key
    gpg: signing failed: No secret key

    I am gonna stop now.
    I will wait for others input.

    Luciano Resende
    @lresende
    let me try to create some “release candidate” by end of the week
    please subscribe to the toree dev mailing list to be notified about rc availability
    V4G4X
    @V4G4X
    Okay
    Thank You
    V4G4X
    @V4G4X

    Okay so I started working on the gpg issue.
    Which after googling around, made me realize that I had no keys under gpg.
    gpg --list-keys hence, was not displaying anything.
    I created keys using gpg --full-generate-key and sudo gpg --full-generate-key both.
    And the

    gpg: no default secret key: No secret key   
    gpg: signing failed: No secret key

    issues disappeared.

    But now they're replaced by:

    GPG is gpg and file is dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz
    gpg: signing failed: Inappropriate ioctl for device
    gpg: signing failed: Inappropriate ioctl for device
    GPG is gpg and file is dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-src/toree-0.5.0.dev1-incubating-src.tar.gz
    gpg: signing failed: Inappropriate ioctl for device
    gpg: signing failed: Inappropriate ioctl for device

    I don't know what this means, but a a couple sites recommended running export GPG_TTY=$(tty).
    It apparently worked like magic for others.
    It didn't here.

    The above error remains constant.

    V4G4X
    @V4G4X

    Regardless of the above error.
    I went into /incubator-toree/dist/toree-pip and checked the SHA512SUM of toree-0.5.0.dev1.tar.gz.
    Then I went into /incubator-toree/dist/toree-src and checked the SHA512SUM of toree-0.5.0.dev1-incubating-src.tar.gz.
    They match with the keys provided.
    So I assume the whole process has finally succeeded.

    But there is still a doubt that bugs me, there's no way these files were built with the exact same keys described.
    We literally changed the source code.

    I am probably missing something very important.
    Any and all feedback is appreciated.

    Okay I'm so stupid,

    Generating signature and sums for /home/v4g4x/incubator-toree/dist/toree-bin/toree-0.5.0.dev1-incubating-bin.tar.gz

    It literally says that it's now generating the sums. I was comparing the keys with the original.

    I'm gonna try installing the kernel via the pip method.
    Fingers Crossed

    V4G4X
    @V4G4X

    This was my first message here, before trying to build toree from source.

    When I open jupyter and when it tries to start the kernel, it never manages to.
    And this is the exception I see occurring in the terminal used to open Jupyter.
    ...

    And now I've come full circle. I still get an exception when running jupyter and using Toree,
    During the beginning when the kernel tries to establish a connection.
    Except now the exception/error has slightly changed:

    [init] error: error while loading Object, Missing dependency 'class scala.native in compiler mirror', required by /modules/java.base/java/lang/Object.class
    
    Failed to initialize compiler: object scala.runtime in compiler mirror not found.
    ** Note that as of 2.8 scala does not assume use of the java classpath.
    ** For the old behavior pass -usejavacp to scala, or if using a Settings
    ** object programmatically, settings.usejavacp.value = true.
    
    Failed to initialize compiler: object scala.runtime in compiler mirror not found.
    ** Note that as of 2.8 scala does not assume use of the java classpath.
    ** For the old behavior pass -usejavacp to scala, or if using a Settings
    ** object programmatically, settings.usejavacp.value = true.
    Exception in thread "main" java.lang.NullPointerException
        at scala.tools.nsc.Global.exitingTyper(Global.scala:1025)
        at scala.tools.nsc.interpreter.IMain$Request.x$19$lzycompute(IMain.scala:853)
        at scala.tools.nsc.interpreter.IMain$Request.x$19(IMain.scala:852)
    .....and more
    V4G4X
    @V4G4X
    I feel SO stupid.

    I did all of this.

    But what solved my problem.
    Was simply switching from Java 11 to Java 8.

    V4G4X
    @V4G4X
    That's it
    It's working now.
    Thanks a lot guys.
    Even though it was my first time, building from source was really fun.
    Hope to see Toree v1.0 soon
    Luciano Resende
    @lresende
    Not java, but Scala…
    Unlike Java, Scala binary compatibility is not so great, so when you move from one version to another you need to recompile, there were also little tweaks need in the code, and all of these has been done in master already, that’s why building from master it works
    Pol Santamaria
    @polsm91

    Hi, we are testing the Apache Toree with Jupyter Enterprise Gateway on k8s. It seems that when the kernel is launched, the dependencies specified by "--packages" are downloaded in the enterpise-gateway and then the driver has no access to it.
    Another solution we tried was using the magicline %AddDeps, then they are downloaded into the /tmp of the Driver pod, but it fails to use them. I think it is related to the SparkContext already being created, since we are trying to load the "hadoop-azure" and similars to access the Azure blob storage.

    Does anybody have some suggestions on how can I solve this dependencies issue? Many thanks! :)

    Kevin Bates
    @kevin-bates
    Hi @polsm91. I would imagine this could be accomplished via hdfs/http references or extending the TOREE_SPARK_OPTS in the kernel.json with volume information: https://spark.apache.org/docs/2.4.6/running-on-kubernetes.html#dependency-management
    Do either of those approaches fit your scenario/environment?
    Pol Santamaria
    @polsm91
    Thanks @kevin-bates for your suggestions. To clarify, what we would like is for Spark to download the packages from Maven. For regular jars (application jars), we are storing them on S3 and calling %AddJar for now, and I don't think is viable to store all the dependencies there.
    When we add the --packages ${MVN_DEPS} to the __TOREE_SPARK_OPTS__ in kernel.json, Spark Toree downloads the dependencies to the enterprise_gateway pod and are forgotten there. Since the gateway serves multiple users, we can't put a shared volume on the $HOME/.ivy2 folder of the gateway.
    Pol Santamaria
    @polsm91
    Another option could be downloading the packages with %AddDeps, and setting a shared volume where they are downloaded. However, in my case, the Spark driver was not detecting the packages providing another FileSystem for Hadoop (I also tried adding the --transitive flag. For example:
    // We run on Spark 2.4.6 compiled for Scala 2.11
    %AddDeps io.delta delta-core_2.11 0.6.1
    %AddDeps org.apache.hadoop hadoop-azure 2.7.3
    %AddDeps com.azure azure-storage-blob 12.8.0
    %AddDeps com.azure azure-storage-common 12.8.0

    Marking io.delta:delta-core_2.11:0.6.1 for download
    Obtained 2 files
    Marking org.apache.hadoop:hadoop-azure:2.7.3 for download
    Obtained 2 files
    Marking com.azure:azure-storage-blob:12.8.0 for download
    Obtained 2 files
    Marking com.azure:azure-storage-common:12.8.0 for download
    Obtained 2 files

    import org.apache.hadoop.fs.azure.NativeAzureFileSystem // Just to verify it can be found
    val df=spark.read.format("delta").load("wasb://######") // This line fails

    java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:160)
    at org.apache.spark.sql.delta.sources.DeltaDataSource$.parsePathIdentifier(DeltaDataSource.scala:252)
    at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:153)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    ... 50 elided
    Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
    ... 64 more

    Pol Santamaria
    @polsm91
    I assume the reason is that the SparkContext was already created when the Azure packages were added, and the Hadoop FileSystem submodule needs to be reloaded in some way :/
    Kevin Bates
    @kevin-bates

    Since I’m not a Spark expert, I can’t tell if you’re running into issues with k8s and spark in general or something with the way EG works. Here are some other pieces of information you might find helpful.

    The image for the ‘vanilla’ Scala and spark Scala kernelspecs is the same image. As a result, you could try launching the vanilla Scala image and configure the spark context yourself to see if that helps, or you can modify the spark scala kernelspec and set --RemoteProcessProxy.spark-context-initialization-mode=None so that it doesn’t create a spark context. The difference between these is that with the former, we control the kernel-pod configuration, whereas Spark controls the pod with the latter and you need to specify pod-related settings in SPARK_OPTS (_TOREE_SPARK_OPTS).

    Also note that any envs prefixed with KERNEL_ flow from the client to EG and are available to the kernel. As mentioned, the vanilla kernels are associated with a kernel-pod.yaml template file (in their scripts directories) that can be altered for things like mounts, init containers, etc. and the user-specific “names” could be specified via a KERNEL_ env value. (Note that KERNEL_USERNAME is already used by many applications for this.)

    In EG 3.0 - which will support Spark 3.0, Spark on Kubernetes enables “pod templates” which will enable the convergence of the two launch approaches (the Spark 2.x on k8s controls the pod creation - so we can’t use our templated approach for that). PR jupyter/enterprise_gateway#559 has been waiting around this - but we’re not there yet.

    Luciano Resende
    @lresende
    Another simple test that could be done is to check what is the Spark behavior when doing this outside of the kernel environment, e.g. when you submit an app passing the opts and see how Spark behave
    Pol Santamaria
    @polsm91
    Many thanks @kevin-bates for the extensive reply. I will experiment with your suggestions and also explore how it is going to work in EG 3.0, we run Spark 2.X but we plan on upgrading to 3.X before 2021.