My guess is there are two different python installations around that are confusing things
one of them is no longer there (maybe partially deleted, given the missing directory)
I’d try reinstalling Python, maybe from homebrew
or fixing your PATH, in case there are multiple installations there
maybe your
python3-config is from a different python installation than your python? which python3-config and which python should point to the same directory
for example, on my machine they are both in
/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/bin/
(I installed python with homebrew)
Nope, I use the virtualenv Python instead, just don't want it messed up local environment. I am trying to use polynote along with my demo on Spark and provide instruction for users to start using it. Using Docker although would work but I am looking for some out-of-box solution for users so they don't have to dive into configuration... Also my application will also use GPU, I am not sure if the docker env will have CUDA driver
yeah... the polynote docker image doesn't support CUDA... I think it's not that hard to build an image that works with CUDA, though (I have a WIP, but my nVidia GPU is on my Windows machine, and getting docker to play well with GPU's in windows is not trivial, so I gave up)
python3-config --ldflags
-bash: python3-config: command not found
Thanks for you all, I got my issue fixed: installing python3 on my laptop. the system default python3 seemed not working well: after following the brew install python3 instruction I managed to fix it: https://stackoverflow.com/questions/49704364/make-python3-as-my-default-python-on-mac
@lanking520 you can just use Scala and python stuff shouldn't even be initialized at all... if you mean "can I install it with no python" then I think it ought to be possible except that the run script is itself python. If you start it directly with
java then I don't see why it wouldn't work (though I haven't tried)
So I didn't even have to install the python dependencies
So here is the thing, I am one of the author of https://github.com/aws-samples/d2l-java this project to allow all Java developers to learn DL in Java. We already onboard with Jupyter Notebook and did several hacks to allow Java runs everywhere within Jupyter Notebook. However, the notebook itself lacks of language feature, such as poping up recommendation functions or help on the grammer issues. I found this project truly helpful to help people leverage Scala. So as a start, I planned to create a short tutorial on Spark with Scala for Deep Learning with PolyNote to see how things goes.
As for running directly without python dependencies, like @jeremyrsmith said, the run script https://github.com/polynote/polynote/blob/master/scripts/polynote.py builds a
java command that lets you run Polynote. If you don’t need Python support, you can just take a look at the output of the script once and then use that to run it next time.
And this works perfectly
#!/usr/bin/env bash
# Standalone Scala usage for PolyNote
jars="polynote.jar"
for f in deps/*.jar
do
jars+=":$f"
done
java -cp $jars polynote.Main
Oh, like for the interpreter's compiler? Yeah, we definitely need to be able to pass compiler flags into there! I feel like we had some way to do this but I can't remember what it was... PR would be welcome for this though, even if it's something simple/non-configurable to start with
(I feel like things like that are OK to start out as like env variables or server arguments, and then can grow into having UI around them... I don't want to stall useful features based on making UI around them)
(or, even config file elements. That's not too bad and doesn't require UI around it)
TBH I think
-target:jvm-1.8 probably ought to be there by default in the non-spark case, because even without Spark I think we still require JVM 1.8+
(still PR welcome :grimacing: )
Do you know if there is any places I can try to inject this flag and test? I can try to build from source and see if it works. If PolyNote require Java8 and above, I think it is worthwhile to add this compiler flag
This is the error thrown:
Error:Static methods in interface require -target:jvm-1.8 (Line 1)
I think it's fair to say that polynote requires Java 8 or above. We certainly don't test it with anything < Java 8. The first thing I would do here is add a notebook config option for "extra scalac flags" (probably called
scalacOptions to align with sbt) that gets parsed into scala.tools.nsc.settings.ScalaSettings (there is a method to do this somewhere...) and gets applied to the default Settings in the ScalaInterpreter
At some point we could also add a UI to edit that, or maybe even a fancy UI for those settings like IntelliJ has (with checkboxes for common things and whatnot)
Actually, maybe just adding
-target:jvm-1.8 as a default setting would be a fine first step.
It wouldn't solve generally fiddling with the scalac flags, but at least would solve this thing and it's pretty easy and non-controversial. @lanking520 that would probably go hereabouts https://github.com/polynote/polynote/blob/master/polynote-kernel/src/main/scala/polynote/kernel/ScalaCompiler.scala#L658 if you want to play with it
(if you would rather it just work, I can make the change... but in case it might be fun for you :smile:)