my setup is tricky because our codebase shares way too much code between our services and Spark jobs
we can get around bin compat issues with sbt-assembly shading rules
but I was a bit tired of moving 100+ MB fat JARs around just to do some exploration with the Spark shell
First i bumped in this issue: almond-sh/almond#508
now seems the kernel it's hanging
is there a way to debug it?
I separated my statements into smaller chunks and now seems to work
weird
I am running almond with docker with:
docker run -it --rm -p 8888:8888 almondsh/almond:latest. How can I access my local file system?
I want to read a file in
/sysroot/home/victor/Documentos/test.csv
@vherasme you can use the
-v option to mount a host directory into your container. See https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount
I was able to create the notebook in the
work directory. Will this notebook disappear once I quit?
Hi All, is there currently a way to install the almond.sh kernel in offline mode? I am unfortunately behind the corporate firewall and cannot install it with suggested in the docs
./coursier launch almond -- --install . I saw the issue on GitHub: almond-sh/almond#145 however the link in the answer is not valid anymore. I'd appreciate tips on how to go about it. Thanks!
@mrjoseph84 You could try to generate a standalone launcher as described in the docs: https://almond.sh/docs/install-other#all-included-launcher
@mrjoseph84 You could try to generate a standalone launcher as described in the docs: https://almond.sh/docs/install-other#all-included-launcher
Thanks! I'll check that.
Hello, is there any way to pass the Spark conf parameters via environment variables so. that you don't need to specify them in the notebook? For PySpark it's
PYSPARK_SUBMIT_ARGS pyspark-shell, for Zeppelin it's SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh (https://zeppelin.apache.org/docs/0.5.5-incubating/interpreter/spark.html). Is there any way to pass the envs in similiar way in Almond kernel (e.g. via some Ammonite configs)?
That doesn't allow to specify the options directly in an environment variable though. But support for it could be added. It should be a matter of adding some logic reading an env var around here.
this is my snippet of code:
import $ivy.`org.apache.spark::spark-sql:2.4.0`
import $ivy.`sh.almond::ammonite-spark:0.5.0`
import org.apache.spark.sql._
val spark = {
AmmoniteSparkSession.builder()
.master("yarn")
.config("spark.executor.instances", "20")
.config("spark.executor.memory", "2g")
.getOrCreate()
}
def sc = spark.sparkContext
val rdd = sc.parallelize(1 to 100000000, 100)
val n = rdd.map(_ + 1).sum()
I use almond:0.5.0 --scala 2.11.12
ammonite:1.6.7
Hi all, I'd like to store snippets of code from a cell in Jupyter directly to a text file. In IPython, one could do that using magic commands (%...). Any pendant to that in Almond? I found some discussions from 2016 stating it's not possible, but maybe things changed in the meantime? Thanks
4 replies
Hi, I'm trying to use the almond kernel with jupyter-book, but it doesn't like the ANSI color codes generated by ammonite. It is supposed to be possible to turn off the colors with
interp.colors() = ammonite.util.Colors.BlackWhite, but that doesn't seem to have any effect from within Jupyter (no errors, it just doesn't turn off the color). It does work from an ammonite REPL. Any ideas? I don't want to file an issue before I know whether this is on the almond side or the ammonite side (and there's already an issue with jupyter-book to handle colors, but I don't think it's going to happen soon).
7 replies
Hi there, I'm importing a script in a notebook through
import $file.script. It looks like that once the script is compiled successfully and imported will no longer be reimported also if the script changes when the cell is newly evaluated. Is there a way to force the compilation or the reimport of the script without restarting the kernel?
2 replies
Hi, it looks like almond is not downloading runtime dependencies of libraries: If I run
import $ivy.`io.grpc:grpc-core:1.30.2`
io.perfmark.PerfMarkI get "object perfmark is not a member of package io", while perfmark should be a runtime dependency for grpc-core, and should be downloaded and put on the classpath as well. Is there some setting to change this behavior?
apologies if this is too deep of a jupyter vs. jupyterlab question.
I have a custom kernel written on top of Almond and it works great in old jupyter notebook.
The main crux is that in execute I start a computation asynchronously and immediately return success:
val id: String = ...
ExecuteResult.Success(
DisplayData.markdown("_awaiting results_").withId(s"$id-output"))And then the async process will use $id-output to render the results as html.
all the other output updates work in jupyterlab apart from rendering the final output. Has anything changed in Almond's/Jupyter's API?
1 reply
How do I fix this?
I added support for this in the Kotlin Jupyter Kernel recently (Kotlin/kotlin-jupyter#102) and have a real use case for this
I am trying to use Spark 3.0 with the local standalone cluster setup. I just simply create 1 master and 1 worker locally. However, the job is keep crashing with the issue
20/11/23 15:55:24 ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to null
...
Caused by: java.io.IOException: No FileSystem for scheme: http
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)It seemed all jars are uploaded to the Spark remote server and Spark trying to fetch them.
import $ivy.`org.apache.spark:spark-sql_2.12:3.0.0`
import org.apache.spark.sql._
val spark = {
NotebookSparkSession.builder()
.master("spark://localhost:7077")
.getOrCreate()
}
spark.conf.getAll.foreach(pair => println(pair._1 + ":" + pair._2))
def sc = spark.sparkContext
val rdd = sc.parallelize(1 to 100000000, 100)
val n = rdd.map(_ + 1).sum()You can reproduce by using the above code (and please create a standalone cluster beforehand
curl -O https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
tar zxvf spark-3.0.0-bin-hadoop2.7.tgz
mv spark-3.0.0-bin-hadoop2.7/ spark
export SPARK_MASTER_HOST=localhost
export SPARK_WORKER_INSTANCES=1
./spark/sbin/start-master.sh
./spark/sbin/start-slave.sh spark://localhost:7077
This is tested on both Cent OS 7 and Mac with JDK 11, Scala 2.12.10, Almond (latest) + Almond Spark(0.10.9)