Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Peter Parente
    @parente
    which uses Girder, another open source kitware project
    https://girder.readthedocs.io/en/stable/
    Ariella Gladstein
    @agladstein
    But, is the data actually transferred? If I have gigs of data on my university HPC or cloud volume, is the data transferred somewhere?
    Peter Parente
    @parente
    Matt should correct me if I'm wrong, but my understanding is that the data is transferred to wherever the jupyter notebook kernel is running (might be local machine, might be on a server) and then some is transferred to the web browser for display. Somewhere in that path (either data server to jupyter kernel location, or kernel location to web browser) it gets downsampled and compressed.
    Peter Parente
    @parente
    has some built in analysis and plotting tools, but you can also export data in common formats to use in notebooks
    Ryan Cooper
    @maptastik
    Open data from Raleigh: http://data-ral.opendata.arcgis.com/
    Joan Pharr
    @changintide_twitter
    Untitled... only 7
    Matt McCormick
    @thewtex
    @agladstein , @parente is correct -- the data is usually on local filesystem where the jupyter kernel is running. But, it is possible to download the data locally via standard https over the network, or if using Dask Arrays, the data can be loaded from the network on demand and never saved to disk.
    Joan Pharr
    @changintide_twitter
    image.png
    Peter Parente
    @parente
    Screen Shot 2018-11-13 at 11.55.00 AM.png
    If Untitled notebooks get you down, you can configure Jupyter to name them something else, like RenameMe!!!!0
    Ariella Gladstein
    @agladstein
    @thewtex so if I have a jupyter kernel running on the same system of where my data is, there is no data transfer happening?
    Matt McCormick
    @thewtex
    @agladstein good question, there is transfer from the python process into your browser --- but, we compression and reduce the data as required :-)
    Ariella Gladstein
    @agladstein
    @thewtex I see. thanks
    Wilson Wooten
    @wilsonwooten
    image.png
    That counts right? lol
    Tony Fast
    @tonyfast
    image.png
    Wilson Wooten
    @wilsonwooten
    image.png
    Eric Dill
    @ericdill
    I'm up to 14
    Screen Shot 2018-11-13 at 1.00.18 PM.png
    Justin Smith
    @jussmi
    $ find . -name 'Untitled*.ipynb' | grep -v 'checkpoint' | wc -l
    129
    :frowning:
    Ariella Gladstein
    @agladstein
    Screen Shot 2018-11-13 at 1.03.40 PM.png
    But, I'm the real winner with 0
    Anthony Schneider
    @AnthonySerious_twitter
    I did some cleanup recently from 14 down to 5. Think I'm out of the running.
    Peter Parente
    @parente
    Screen Shot 2018-11-13 at 1.08.16 PM.png
    ... if i lay claim to all 1393 Untitled* notebooks in one of our JupyterHub environments
    Andre Violante
    @aviolante
    @agladstein yeah I can see that. I'm still a huge fan of R Studio. Best of luck!
    Ariella Gladstein
    @agladstein
    Can code ocean be connected to github so it is updated with commits on github? And how can it be integrated with large data? And where does it run? Suppose, hypothetically, I publish a paper that uses 1T data and train large memory deep learning GPU models on it... can I use code ocean to make that analysis reproducible?
    Greg Frazier
    @gnfrazier
    My presentation notebook is on github https://github.com/gnfrazier/jupyter-day-presentation
    Andre Violante
    @aviolante
    Anthony Schneider
    @AnthonySerious_twitter
    I assume this is the non-blocking cell magic? https://gist.github.com/josh-howes/50e53f3ad52e11b2bc85fb6964147107
    Peter Parente
    @parente
    @/all Feel free to link presentation materials from your talk titles here https://github.com/libcce/TriangleJupyter/blob/master/information.md. Anyone with a GitHub account can click the pencil icon and submit a pull request with the link.
    Seth Ariel Green
    @setgree
    @agladstein To put answers to these questions in writing:
    1) Code Ocean currently imports from GitHub but we are aiming for bi-directional links (pull & push) for early 2019.
    2) large data can be uploaded as public datasets https://help.codeocean.com/getting-started/uploading-code-and-data/using-public-datasets -- and details on our machines are here https://help.codeocean.com/faq/what-machine-will-my-code-run-on . Future plans: A) link up with local university clusters B) get the most powerful machines available to run code
    3) Currently we run on AWS.
    4) 1 TB of data creation may be very slow on Code Ocean or exceed system limits -- it's hard to say in advance. It depends on how much memory the capsule uses during runtime. We recommend clearing intermediate objects out of memory early and often. One of our 2019 priorities is more powerful machines intended to help use cases like this.
    Tony Fast
    @tonyfast
    RISE notebook presentations https://github.com/damianavila/RISE
    Ariella Gladstein
    @agladstein
    @setgree Thanks
    Seth Ariel Green
    @setgree
    Code Ocean presentation https://github.com/setgree/jupyter-day-presentation -- don't tell anyone I wrote it in Rmarkdown :grimacing:
    Will Eaton
    @wseaton
    hey, Rmarkdown has nice diffs
    no shame
    Peter Parente
    @parente
    the VSCode Python extension I mentioned uses a flat text file view to represent notebooks and cells with %% prefixes. i haven't played with it yet to see if it can do bi-directional transform between the ipynb JSON and the flat-file format.
    not the answer to better version control of notebooks, but in case folks haven't seen it https://nbdime.readthedocs.io/en/stable/ which is also being wrapped as a jupyterlab extension
    Will Eaton
    @wseaton
    related for vim users: https://github.com/szymonmaszke/vimpyter uses the same paradigm, converting JSON to flat-file and back and forth for rendering in a buffer (doesn't support live update though)
    Seth Ariel Green
    @setgree
    @parente Thanks, nbdime is new to me -- and will be of interest to my teammates as we continue integrating JL-features :pray: but wait of course they already knew it
    Tony Fast
    @tonyfast
    nbdime is 👌 in lab
    00
    Joan Pharr
    @changintide_twitter
    Got a question during the coffee break about jupyter notebooks rendering in bitbucket, the answer is yes : https://bitbucket.org/site/master/issues/11318/enable-rich-rendering-of-jupyter-notebooks
    Peter Parente
    @parente