polynote/polynote

Brad Frost

@bradfrosty

I don't believe another service is bound on that interface as they are different IPs (one for the pod itself, the other for the service)

Yes, let me try the kernel host parameter to be the driver pods IP

is it possible to use an env var for these config params?

jonathanindig

@jonathanindig

No, but you could use envsubst to make that easy if you dont wanna mess with sed :) https://stackoverflow.com/a/14157575

Brad Frost

@bradfrosty

Hmm... so I'm mounting the config from a kubernetes configmap to /opt/config/config.yml... so it's kind of a chicken and egg as to getting the pod IP (the polynote server needs the config, but the config needs the polynote server IP)

jonathanindig

@jonathanindig

for example:

# this is /opt/polynote.conf.template
listen:
    host: 0.0.0.0
    port: 8192

kernel:
    host: ${HOST_IP}
    port_range: ${KERNEL_PORTS}

And then in your server container launch script, before launching the server
envsubst '${HOST_IP} ${KERNEL_PORTS}' < /opt/polynote.conf.template > /opt/polynote.conf

(where HOST_IP and KERNEL_PORTS are environment variables available to your run script)

Brad Frost

@bradfrosty

ah got it

ok trying that

btw, thanks again for the help :)

jonathanindig

@jonathanindig

no worries :) sorry for leading you down the wrong path earlier :sweat_smile:

Brad Frost

@bradfrosty

Hey @jonathanindig following up. Everything worked after setting the kernel.listen IP to the IP of the polynote server.

I was able to get the hostname working after that, just needed to use the right kubernetes service configuration. Thanks for the help!

jonathanindig

@jonathanindig

Glad to hear it! Thanks for following up :smile:

Anton Kulaga

@antonkulaga

I see that latest release was in September, any plans to make a new one?

larsskaug

@larsskaug

polynote.py run filename.py hangs. What am I missing? Has anyone been able to run a notebook in "headless" mode?

jonathanindig

@jonathanindig

Hi @larsskaug, I see you posted an issue a while ago about this. The notebook runner was never documented, it was more of a POC; it’s possible that it broke somehow during a refactor since last May

We do plan on solidifying the story around this feature eventually (see the overarching issue I created about this last week: polynote/polynote#1052 )

jonathanindig

@jonathanindig

All that being said, I just tried it and it worked for me :smile: so a little more information about your environment would be helpful.

I see you did not provide any more information in your issue. Does running the Polynote server work in your environment?

jeremyrsmith

@jeremyrsmith

@larsskaug did you run it with python 3? It requires python 3 (as does Polynote in general). And Polynote probably has to be installed properly for it to work.

Seth Lasky

@TheMover

Hey all, I'm having a little trouble with bringing my library into polynote and using it and hoping someone here could point me in the right direction, I have a trait in an imported library that takes a higher kinded type parameter:

trait Client[F[_]]{

}

but when importing it into polynote and attempting to extend it like:

import cats.effect.IO
object SomeApp extends Client[IO]{
}

and run the program I get the error: my.library.Client does not take type parameters

I also notice that when I define the trait inside polynote instead of importing and attempt to extend it with the imported IO type it gives me a similar error but about IO:
cats.effect.IO takes no type parameters, expected: one

Is this a known issue or is there something I need to change version wise to be able to import types with higher kinded parameters?

Seth Lasky

@TheMover

Both libraries are using scala 2.13 btw

jeremyrsmith

@jeremyrsmith

@TheMover Polynote doesn't support Scala 2.13 yet :( it will be a couple of months for that (we have to drop 2.11 first, but we can't because of Spark)

Seth Lasky

@TheMover

Gotcha, thank you

I changed the scala version to 2.12 but still getting some weirdness, I'm trying to import io.circe and get a classdefnotfound for io.circe.Json. Are there any known issues/conflicts with the circe library?

Seth Lasky

@TheMover

The exact dependencies I'm using are io.circe:circe-core_2.12:0.13.0, io.circe:circe-generic_2.12:0.13.0, io.circe:circe-parser_2.12:0.13.0. The notebook looks like:

import io.circe.parser.parse
import io.circe.generic.auto._

val json = """{"some":"json", "other":"stuff"}"""
val parsedJson = parse(json)

and the error message is:

     Uncaught exception: scala.Product.$init$(Lscala/Product;)V (java.lang.NoSuchMethodError)

        io.circe.Json.<init>(Json.scala:16)
        io.circe.Json$JNull$.<init>(Json.scala:265)
        io.circe.Json$JNull$.<clinit>(Json.scala:-1)
        io.circe.Json$.<init>(Json.scala:451)
        io.circe.Json$.<clinit>(Json.scala:-1)
        io.circe.jawn.CirceSupportParser$UnlimitedFacade.jstring(CirceSupportParser.scala:100)
        io.circe.jawn.CirceSupportParser$UnlimitedFacade$$anon$8.add(CirceSupportParser.scala:110)
        org.typelevel.jawn.CharBasedParser.parseString(CharBasedParser.scala:96)
        org.typelevel.jawn.CharBasedParser.parseString$(CharBasedParser.scala:93)
        org.typelevel.jawn.StringParser.parseString(StringParser.scala:15)
        org.typelevel.jawn.Parser.rparse(Parser.scala:445)
        org.typelevel.jawn.Parser.parseTop(Parser.scala:380)
        org.typelevel.jawn.Parser.parse(Parser.scala:364)
        org.typelevel.jawn.SyncParser.parse(SyncParser.scala:23)
        org.typelevel.jawn.SupportParser.$anonfun$parseFromString$1(SupportParser.scala:15)
        ......

jeremyrsmith

@jeremyrsmith

@TheMover currently there is a separate polynote distribution for 2.11 and 2.12. This will change in the next release, but for now make sure you're using the right one for the dependencies you want to use.

Seth Lasky

@TheMover

That makes sense, switching to latest circe version for 2.11 seemed to resolve things, I downloaded polynote via brew as a shortcut, is there a way to grab it as the 2.12 distribution or should I just download direct from site

jonathanindig

@jonathanindig

TIL polynote was on homebrew…

Seth Lasky

@TheMover

LOL I always attempt homebrew as the easy road and it always ends up making things more confusing

jonathanindig

@jonathanindig

I’d recommend downloading the distribution tarball from our releases page: https://github.com/polynote/polynote/releases/tag/0.3.12

People just put stuff on homebrew I guess…

Seth Lasky

@TheMover

Doing that now, appreciate the quick responses!

antonkulaga

@antonkulaga:matrix.org

[m]

@jonathanindig: BTW, do you plan to make a new polynote release soon? I see there have been many improvements since September

jonathanindig

@jonathanindig

Yes, we plan on releasing soon.

antonkulaga

@antonkulaga:matrix.org

[m]

Thats is great as I am a bit worried about releases. I switched 3 notebooks before: scala notebook->apache toree -> zeppeling -> polynote and for toree and zeppelin it was because at some point they moved to very slow release schedule and many issues were fixed only in nightly versions

Seth Lasky

@TheMover

Is there any way to log from other cells than the one you're running? Even a println statement

jeremyrsmith

@jeremyrsmith

@TheMover like, you have a function iLogStuff defined in one cell, and it calls println, and you want that println to show up in the output of the cell which calls iLogStuff instead of the cell that defines iLogStuff?

With println it ought to just work... with kernel.display you'd need iLogStuff to take an implicit KernelRuntime so that it gets the one from the calling cell

Or, maybe I misunderstood your question... feel free to clarify :smile:

Seth Lasky

@TheMover

The println statements I was making in a higher cell, e.g. iLogStuff weren't showing up in the output of the cell I was calling iLogStuff from

The really weird part was that I think they showed up like 15 outputs later

After I had removed the println

larsskaug

@larsskaug

@jeremyrsmith: I spend 4-8 hours a day in Polynote at this point and it works beautifully. Running it in headless mode does not work. And yes, Python3 is my default python.

larsskaug

@larsskaug

@jonathanindig: I might be unusual due to my combination of (1) being a heavy user of Polynote on (2) Windows. I use both WSL1 as well as WSL2, the latter of which work swimmingly for this purpose. In fact, compared to a straight Ubuntu install on the same hardware, WSL 2 performs within 90%+ even for I/O. This might explain why the headless notebook runner does not work. Here's a log of what happens when I try (It just hangs for more than a half hour after what you see here. The notebook simply contains println("Hello")):

<name>@<server name redacted>$ polynote.py run HelloWorld.ipynb
['java', '-cp', 'polynote.jar:polynote.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-spark-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-collection-compat_2.12-2.1.1.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-compiler.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-library.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-reflect.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-xml_2.12-1.2.0.jar:', '-Djava.library.path=/home/lars/.local/lib/python3.8/site-packages/jep', 'polynote.Main', 'run', 'HelloWorld.ipynb']
[INFO]   Loading configuration from config.yml
[INFO]   Deploying with command:
   |     /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp polynote.jar:polynote.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-run
time.jar:/mnt/c/Users/<username>/Development/polynote/deps/polynote-spark-runtime.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-collection-compat_2.12-2.1.1.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-compiler.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-library.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-reflect.jar:/mnt/c/Users/<username>/Development/polynote/deps/scala-xml_2.12-1.2.0.jar: -Djava.library.path=/home/lars/.local/lib/python3.8/site-packages/jep polynote.kernel.remote.RemoteKernelClient --address 127.0.0.1 --port 57471 --kernelFactory polynote.kernel.LocalKernelFactory

jeremyrsmith

@jeremyrsmith

@larsskaug thanks for the logs... I don't have a Windows 10/WSL machine to test on, but I'll take a look at the logs at least and see if anything stands out. Certainly it seems like it ought to work under WSL (it's Java after all... cross platform right?)

@larsskaug so you don't get any further logs? What if you make a notebook that throws an exception instead?

jeremyrsmith

@jeremyrsmith

On linux and mac, it seems to work, which makes it all the more distressing that it doesn't work on WSL – though tbh we haven't really been working on or testing the headless runner, and even if we did it would likely be mostly useless – because it can't do plots (because plots require the front-end) and plots are probably the #1 use case of the headless runner :disappointed:

Where communities thrive

People

Repo info

Activity