Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Adrian Petrescu
    Testing the new room
    Alejandro Guerrero Gonzalez
    hello @apetresc!! :)
    Adrian Petrescu
    Hey @aggFTW :)
    You found the channel! I just set it up a few days ago
    Just saw your e-mail, kicked off the job again
    Okay beautiful, that worked
    I'll do one last round of manual testing from test.pypi.org and push to release tonight
    I'm actually in a hotel room for Microsoft Ignite as we speak :P
    Alejandro Guerrero Gonzalez
    haha nice!
    i did find the channel through the commit
    good luck with the conference!
    let me know if you need more help!
    Hi all
    Did anyone use spark magic with Aws Emr ?
    Adrian Petrescu
    Hey @abhisam - yes, I do that frequently :)
    Adrian Petrescu
    Are you encountering any issues?
    Are you able to plot any graphs in a Emr with jupyerhub application ?
    I mean using Matplotlib package ...
    Hi guys
    Joaquín Chemile
    Hello Guys!! Do you have any resourse where i can find the way to import my scala local functions to the Notebook (Something like when i don "Pip install ." whith Python) - Thanks!
    Javier Bianco
    Hi guys, do you know how can I select a different sparkmagic "config.json" file when a kernel is starting? I've tried modifying the "kernel.json" file adding two variables into "env" secction: "env":{"SPARKMAGIC_CONF_DIR":"/home/basic/.sparkmagic", "SPARKMAGIC_CONF_FILE":"config02.json"} but didnt work. Thanks!
    John Pugliesi

    Hello - I'm using sparkmagic to execute long-running spark jobs, and running into an issue where the sparkmagic livy session dies if the submitted spark job runs longer than the configured livy.server.session.timeout. The error manifests as the usual:

    An error was encountered:
    Invalid status code '404' from http://my-livy-server:8998/sessions/38 with error payload: {"msg":"Session '38' not found."}

    Does the sparkmagic session heartbeat thread not keep the session alive if a cell runs longer than the livy session's timeout?

    Appreciate the help

    Itamar Turner-Trauring
    woah I did not realize this existed
    I guess it's rarely used so maybe that's OK
    but in general—if there's no one here and you have a problem, please file an issue on GitHub
    Hey Team, I am trying to figure out a way in sparkmagic to supply additional jars after session creation.

    i tried using:

    %%configure -f
    {"conf": {
        "spark.jars.packages": "com.xyz.my-jar:0.1.1"}

    But this errors out with message: ERROR: Cell magic %%configure not found

    can someone guide me what to do in this case, i am using jupyternotebook
    @here ^^
    Palaniappan Nagarajan
    HI Team, I have a question about configuring spark magic in notebook programatically. I am using PaperMill to execute my notebook and I want to parameterize notebook such that I can execute notebook in different url end points (overriding kernel_python_credentials in ~/.spark/config.json). Is there a way to accomplish?
    Any help or pointers would be greatly helpful. Thanks!
    @here : any examples on configuring default endpoints ? Saw it from the codebase that there is a way to configure default endpoints without the user having to go through the widget. Any pointers in this would be helpful ?

    Hi, I am using for first time spark magic,

    I am in a good position now, as it connects with Livy, but I have a problem. No session is created:

    I execute :

    %load_ext sparkmagic.magics

    And the output is not a javascript widget, it is just

    MagicsControllerWidget(children=(Tab(children=(ManageSessionWidget(children=(HTML(value='<br/>'), HTML(value='…

    I have verified that pip install ipywidgets is installed and

    jupyter nbextension enable --py --sys-prefix widgetsnbextension
    Enabling notebook extension jupyter-js-widgets/extension...
          - Validating: OK

    Using the latest jupyterhub image.

    Then if I run :

    %%configure -f
    {"executorMemory": "1000M", "executorCores": 4}
    Current session configs: {'executorMemory': '1000M', 'executorCores': 4, 'kind': 'spark'}
    No active sessions.

    There are no active sessions.

    Not sure what it is missing. I am getting out of ideas. Any idea? Thanks a lot