by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 13:50
    acse-fk4517 commented #595
  • Jul 01 15:56

    alexarchambault on gh-pages

    Update website (compare)

  • Jul 01 15:47

    alexarchambault on gh-pages

    Update website (compare)

  • Jul 01 15:04

    alexarchambault on master

    Add gitpod config (#605) (compare)

  • Jul 01 15:04
    alexarchambault closed #605
  • Jul 01 14:28

    alexarchambault on gh-pages

    Update website (compare)

  • Jul 01 14:01
    alexarchambault synchronize #605
  • Jul 01 13:53
    alexarchambault synchronize #605
  • Jul 01 13:49
    alexarchambault opened #605
  • Jul 01 13:44

    alexarchambault on master

    Change "export as script" exten… Ignore closed channel exception… Merge pull request #604 from al… (compare)

  • Jun 30 18:14
    alexarchambault synchronize #604
  • Jun 30 18:06
    alexarchambault opened #604
  • Jun 30 16:33

    alexarchambault on gh-pages

    Update website (compare)

  • Jun 30 15:45

    alexarchambault on master

    Tweak heartbeat handling (#603)… (compare)

  • Jun 30 15:45
    alexarchambault closed #603
  • Jun 30 15:28
    alexarchambault opened #603
  • Jun 30 13:55

    alexarchambault on v0.10.1

    (compare)

  • Jun 30 13:33

    alexarchambault on gh-pages

    Update website (compare)

  • Jun 30 13:27

    alexarchambault on gh-pages

    Update website (compare)

  • Jun 30 13:13

    alexarchambault on master

    Update case-app, case-app-annot… (compare)

Henry
@hygt
so maybe I should publish my patched version of Spark to our local artifactory and let AmmoniteSparkSession fetch the jars
Alfonso Roa
@alfonsorr
can you show the imports and the way that you create the SparkSession'
and be sure to use a 2.11 kernel and spark version 2.3.2
Henry
@hygt
I'm using scala 2.12, spark 2.4.4 and ammonite-spark 0.9.0
I was trying something like that
import $ivy.`sh.almond::ammonite-spark:0.9.0`
import ammonite.ops._

val sparkHome = sys.env("SPARK_HOME")
val sparkJars = ls ! Path(sparkHome) / 'jars
interp.load.cp(sparkJars)

@

import org.apache.spark.sql._

val spark =
  AmmoniteSparkSession
    .builder()
    .loadConf("my.conf")
    .progressBars()
    .master("yarn")
    .getOrCreate()
but I always ran in some issue or another
I published my patched Spark to our artifactory, unset SPARK_HOME and it works much better
Henry
@hygt
ok now I have another issue, shell works but notebooks not so much :grimacing:
Creating SparkSession
Error initializing SparkContext.
java.io.IOException: No FileSystem for scheme: jar
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
 ...
using the same configuration file
Henry
@hygt
OK, I found a workaround: alexarchambault/ammonite-spark#118
now I have some binary compatibility issue with Jackson even though all jars in my session look aligned :grimacing:
com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.3 requires Jackson Databind version >= 2.10.0 and < 2.11.0
  com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
...
org.apache.spark.sql.almondinternals.NotebookSparkSessionBuilder.getOrCreate(NotebookSparkSessionBuilder.scala:62)
and my jars:
List(
  "/jackson-module-paranamer-2.10.3.jar",
  "/jackson-module-scala_2.12-2.10.3.jar",
  "/jackson-databind-2.10.3.jar",
  "/jackson-annotations-2.10.3.jar",
  "/jackson-core-2.10.3.jar",
  "/jackson-core-2.10.1.jar",
  "/jackson-datatype-jdk8-2.10.1.jar",
  "/jackson-databind-2.10.1.jar",
  "/jackson-datatype-jsr310-2.10.1.jar",
  "/jackson-annotations-2.10.1.jar",
  "/jackson-annotations-2.10.3.jar",
  "/jackson-databind-2.10.3.jar",
  "/jackson-module-paranamer-2.10.3.jar",
  "/jackson-module-scala_2.12-2.10.3.jar",
  "/jackson-core-2.10.3.jar"
)
Henry
@hygt
ha I think I understand why, jackson is in the almond launcher assembly! ok I know how to fix this...
sorry about all my rambling this is slowly driving me crazy
Wojtek Pituła
@Krever
Does any of you guys maybe have a way of retrieving notebook filename being run? Not that this is specific to almond but most of the solutions on the internet use python, so want to ask you first.
Andrew
@sheerluck

@Krever

%%javascript
var kernel = IPython.notebook.kernel;
var thename = window.document.getElementById("notebook_name").innerHTML;
var command = "theNotebook = " + "'"+thename+"'";
kernel.execute(command);

Wojtek Pituła
@Krever
Awesome, thanks!
I wanted to hide it in my scala lib, but will figure sth out...
Sören Brunk
@sbrunk
@Krever you totally can. Here’s the almond version:
kernel.publish.js("""
  var thename = window.document.getElementById("notebook_name").innerHTML;
  var command = 'val theNotebook = "'+thename+'"';
  Jupyter.notebook.kernel.execute(command);
""")
image.png
Note that it only works in the classic notebook, not in JupyterLab due to security restrictions
Sören Brunk
@sbrunk
But you can put it into a library by adding scala-kernel-api as a provided dependency as described in the docs. https://almond.sh/docs/api-access-instances#from-a-library
@hygt Does it work for you now? Anyway, thanks for sharing what you’ve found trying to solve these issues.
Wojtek Pituła
@Krever
@sbrunk thanks, didnt know such things are possible. I will have to figure out sth that works in Jupyter lab, but now I have all the pieces on almond side.
Henry
@hygt
@sbrunk yes I've gotten to a point where it works. I've built the Almond launcher with coursier's --assembly-rule exclude-pattern ... to get rid of Jackson and Json4s classes. No more binary compatibility issues
my setup is tricky because our codebase shares way too much code between our services and Spark jobs
we can get around bin compat issues with sbt-assembly shading rules
but I was a bit tired of moving 100+ MB fat JARs around just to do some exploration with the Spark shell
Henry
@hygt
also our data scientists would rather use notebooks :smiley:
Pedro Larroy
@larroy
is almond working? I'm trying to run it in jupyter notebook and finding all kinds of problems
First i bumped in this issue: almond-sh/almond#508
now seems the kernel it's hanging
is there a way to debug it?
I separated my statements into smaller chunks and now seems to work
weird
Wojtek Pituła
@Krever
how would you approach rendering basic grapgh diagram? just some nodes and edges
Wojtek Pituła
@Krever
@alexarchambault I see that in master ammonite is in ver 2.0.4 but cs resolve -t sh.almond:scala-kernel_2.13.1:0.9.1 shows it in 1.7.4. Could we have a release with the newer version?
Sören Brunk
@sbrunk
@Krever Alex has just released 0.10.0 which updates ammonite to 2.1.4
Wojtek Pituła
@Krever
Great, thanks for the ping!
Victor M
@vherasme
Hello People
I am running almond with docker with: docker run -it --rm -p 8888:8888 almondsh/almond:latest. How can I access my local file system?
I want to read a file in /sysroot/home/victor/Documentos/test.csv
Sören Brunk
@sbrunk
@vherasme you can use the -v option to mount a host directory into your container. See https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount
Sören Brunk
@sbrunk
docker run -it --rm -v /sysroot/home/victor/Documentos/:/home/jovyan/data -p 8888:8888 almondsh/almond:latestshould work for you
Victor M
@vherasme
Thanks a lot. One last thing, it won't allow me to create new notebooks: An error occurred while creating a new notebook. Permission denied: data/Untitled.ipynb
I was able to create the notebook in the work directory. Will this notebook disappear once I quit?
Sören Brunk
@sbrunk
Yes only the data directory will persist. You should check the write permissions of your documents dir. Or you could also mount another directory too.
Victor M
@vherasme

Yes only the data directory will persist. You should check the write permissions of your documents dir. Or you could also mount another directory too.

Thanks a lot. It's all working now