Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Brad Frost
    @bradfrosty
    I don't believe another service is bound on that interface as they are different IPs (one for the pod itself, the other for the service)
    Yes, let me try the kernel host parameter to be the driver pods IP
    is it possible to use an env var for these config params?
    jonathanindig
    @jonathanindig
    No, but you could use envsubst to make that easy if you dont wanna mess with sed :) https://stackoverflow.com/a/14157575
    Brad Frost
    @bradfrosty
    Hmm... so I'm mounting the config from a kubernetes configmap to /opt/config/config.yml... so it's kind of a chicken and egg as to getting the pod IP (the polynote server needs the config, but the config needs the polynote server IP)
    jonathanindig
    @jonathanindig

    for example:

    # this is /opt/polynote.conf.template
    listen:
        host: 0.0.0.0
        port: 8192
    
    kernel:
        host: ${HOST_IP}
        port_range: ${KERNEL_PORTS}

    And then in your server container launch script, before launching the server
    envsubst '${HOST_IP} ${KERNEL_PORTS}' < /opt/polynote.conf.template > /opt/polynote.conf

    (where HOST_IP and KERNEL_PORTS are environment variables available to your run script)
    Brad Frost
    @bradfrosty
    ah got it
    ok trying that
    btw, thanks again for the help :)
    jonathanindig
    @jonathanindig
    no worries :) sorry for leading you down the wrong path earlier :sweat_smile:
    Brad Frost
    @bradfrosty

    Hey @jonathanindig following up. Everything worked after setting the kernel.listen IP to the IP of the polynote server.

    I was able to get the hostname working after that, just needed to use the right kubernetes service configuration. Thanks for the help!

    jonathanindig
    @jonathanindig
    Glad to hear it! Thanks for following up :smile:
    Anton Kulaga
    @antonkulaga
    I see that latest release was in September, any plans to make a new one?
    larsskaug
    @larsskaug
    polynote.py run filename.py hangs. What am I missing? Has anyone been able to run a notebook in "headless" mode?
    jonathanindig
    @jonathanindig
    Hi @larsskaug, I see you posted an issue a while ago about this. The notebook runner was never documented, it was more of a POC; it’s possible that it broke somehow during a refactor since last May
    We do plan on solidifying the story around this feature eventually (see the overarching issue I created about this last week: polynote/polynote#1052 )
    jonathanindig
    @jonathanindig
    All that being said, I just tried it and it worked for me :smile: so a little more information about your environment would be helpful.
    I see you did not provide any more information in your issue. Does running the Polynote server work in your environment?
    jeremyrsmith
    @jeremyrsmith
    @larsskaug did you run it with python 3? It requires python 3 (as does Polynote in general). And Polynote probably has to be installed properly for it to work.
    Seth Lasky
    @TheMover

    Hey all, I'm having a little trouble with bringing my library into polynote and using it and hoping someone here could point me in the right direction, I have a trait in an imported library that takes a higher kinded type parameter:

    trait Client[F[_]]{
    
    }

    but when importing it into polynote and attempting to extend it like:

    import cats.effect.IO
    object SomeApp extends Client[IO]{
    }

    and run the program I get the error: my.library.Client does not take type parameters

    I also notice that when I define the trait inside polynote instead of importing and attempt to extend it with the imported IO type it gives me a similar error but about IO:
    cats.effect.IO takes no type parameters, expected: one
    Is this a known issue or is there something I need to change version wise to be able to import types with higher kinded parameters?
    Seth Lasky
    @TheMover
    Both libraries are using scala 2.13 btw
    jeremyrsmith
    @jeremyrsmith
    @TheMover Polynote doesn't support Scala 2.13 yet :( it will be a couple of months for that (we have to drop 2.11 first, but we can't because of Spark)
    Seth Lasky
    @TheMover
    Gotcha, thank you
    I changed the scala version to 2.12 but still getting some weirdness, I'm trying to import io.circe and get a classdefnotfound for io.circe.Json. Are there any known issues/conflicts with the circe library?
    Seth Lasky
    @TheMover

    The exact dependencies I'm using are io.circe:circe-core_2.12:0.13.0, io.circe:circe-generic_2.12:0.13.0, io.circe:circe-parser_2.12:0.13.0. The notebook looks like:

    import io.circe.parser.parse
    import io.circe.generic.auto._
    
    val json = """{"some":"json", "other":"stuff"}"""
    val parsedJson = parse(json)

    and the error message is:

         Uncaught exception: scala.Product.$init$(Lscala/Product;)V (java.lang.NoSuchMethodError)
    
            io.circe.Json.<init>(Json.scala:16)
            io.circe.Json$JNull$.<init>(Json.scala:265)
            io.circe.Json$JNull$.<clinit>(Json.scala:-1)
            io.circe.Json$.<init>(Json.scala:451)
            io.circe.Json$.<clinit>(Json.scala:-1)
            io.circe.jawn.CirceSupportParser$UnlimitedFacade.jstring(CirceSupportParser.scala:100)
            io.circe.jawn.CirceSupportParser$UnlimitedFacade$$anon$8.add(CirceSupportParser.scala:110)
            org.typelevel.jawn.CharBasedParser.parseString(CharBasedParser.scala:96)
            org.typelevel.jawn.CharBasedParser.parseString$(CharBasedParser.scala:93)
            org.typelevel.jawn.StringParser.parseString(StringParser.scala:15)
            org.typelevel.jawn.Parser.rparse(Parser.scala:445)
            org.typelevel.jawn.Parser.parseTop(Parser.scala:380)
            org.typelevel.jawn.Parser.parse(Parser.scala:364)
            org.typelevel.jawn.SyncParser.parse(SyncParser.scala:23)
            org.typelevel.jawn.SupportParser.$anonfun$parseFromString$1(SupportParser.scala:15)
            ......
    jeremyrsmith
    @jeremyrsmith
    @TheMover currently there is a separate polynote distribution for 2.11 and 2.12. This will change in the next release, but for now make sure you're using the right one for the dependencies you want to use.
    Seth Lasky
    @TheMover
    That makes sense, switching to latest circe version for 2.11 seemed to resolve things, I downloaded polynote via brew as a shortcut, is there a way to grab it as the 2.12 distribution or should I just download direct from site
    jonathanindig
    @jonathanindig
    TIL polynote was on homebrew…
    Seth Lasky
    @TheMover
    LOL I always attempt homebrew as the easy road and it always ends up making things more confusing
    jonathanindig
    @jonathanindig
    I’d recommend downloading the distribution tarball from our releases page: https://github.com/polynote/polynote/releases/tag/0.3.12
    People just put stuff on homebrew I guess…
    Seth Lasky
    @TheMover
    Doing that now, appreciate the quick responses!
    antonkulaga
    @antonkulaga:matrix.org
    [m]
    @jonathanindig: BTW, do you plan to make a new polynote release soon? I see there have been many improvements since September
    jonathanindig
    @jonathanindig
    Yes, we plan on releasing soon.
    antonkulaga
    @antonkulaga:matrix.org
    [m]
    Thats is great as I am a bit worried about releases. I switched 3 notebooks before: scala notebook->apache toree -> zeppeling -> polynote and for toree and zeppelin it was because at some point they moved to very slow release schedule and many issues were fixed only in nightly versions
    Seth Lasky
    @TheMover
    Is there any way to log from other cells than the one you're running? Even a println statement
    jeremyrsmith
    @jeremyrsmith
    @TheMover like, you have a function iLogStuff defined in one cell, and it calls println, and you want that println to show up in the output of the cell which calls iLogStuff instead of the cell that defines iLogStuff?
    With println it ought to just work... with kernel.display you'd need iLogStuff to take an implicit KernelRuntime so that it gets the one from the calling cell
    Or, maybe I misunderstood your question... feel free to clarify :smile:
    Seth Lasky
    @TheMover
    The println statements I was making in a higher cell, e.g. iLogStuff weren't showing up in the output of the cell I was calling iLogStuff from
    The really weird part was that I think they showed up like 15 outputs later
    After I had removed the println
    larsskaug
    @larsskaug
    @jeremyrsmith: I spend 4-8 hours a day in Polynote at this point and it works beautifully. Running it in headless mode does not work. And yes, Python3 is my default python.
    larsskaug
    @larsskaug
    @jonathanindig: I might be unusual due to my combination of (1) being a heavy user of Polynote on (2) Windows. I use both WSL1 as well as WSL2, the latter of which work swimmingly for this purpose. In fact, compared to a straight Ubuntu install on the same hardware, WSL 2 performs within 90%+ even for I/O. This might explain why the headless notebook runner does not work. Here's a log of what happens when I try (It just hangs for more than a half hour after what you see here. The notebook simply contains println("Hello")):
    <name>@<server name redacted>$ polynote.py run HelloWorld.ipynb ['java', '-cp', 'polynote.jar:polynote.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-spark-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-collection-compat_2.12-2.1.1.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-compiler.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-library.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-reflect.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-xml_2.12-1.2.0.jar:', '-Djava.library.path=/home/lars/.local/lib/python3.8/site-packages/jep', 'polynote.Main', 'run', 'HelloWorld.ipynb'] [INFO] Loading configuration from config.yml [INFO] Deploying with command: | /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp polynote.jar:polynote.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-run time.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-spark-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-collection-compat_2.12-2.1.1.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-compiler.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-library.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-reflect.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-xml_2.12-1.2.0.jar: -Djava.library.path=/home/lars/.local/lib/python3.8/site-packages/jep polynote.kernel.remote.RemoteKernelClient --address 127.0.0.1 --port 57471 --kernelFactory polynote.kernel.LocalKernelFactory
    jeremyrsmith
    @jeremyrsmith
    @larsskaug thanks for the logs... I don't have a Windows 10/WSL machine to test on, but I'll take a look at the logs at least and see if anything stands out. Certainly it seems like it ought to work under WSL (it's Java after all... cross platform right?)
    @larsskaug so you don't get any further logs? What if you make a notebook that throws an exception instead?
    jeremyrsmith
    @jeremyrsmith
    On linux and mac, it seems to work, which makes it all the more distressing that it doesn't work on WSL – though tbh we haven't really been working on or testing the headless runner, and even if we did it would likely be mostly useless – because it can't do plots (because plots require the front-end) and plots are probably the #1 use case of the headless runner :disappointed: