shadaj/scalapy

lasimon

@lasimon

python3-config --ldflags
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ cd /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ ls
config.c install-sh libpython3.6m-pic.a libpython3.6.so makesetup python.o Setup.config
config.c.in libpython3.6m.a libpython3.6m.so Makefile python-config.py Setup Setup.local

Shadaj Laddad

@shadaj

Ah, looks like you have Python 3.6, which ScalaPy doesn't load by default (though perhaps it should). For now, could you try adding the environment variable SCALAPY_PYTHON_LIBRARY=python3.6 before launching SBT?

lasimon

@lasimon

larrysimon@inspiron:~/IdeaProjects/wumpus$ echo $SCALAPY_PYTHON_LIBRARY
python3.6
then sbt run
... same error

BTW what I'm doing is a reference implementation of a solution to the classic Russell & Norvig Wumpus World using Deep RL on pyTorch. A Python version is an assignment I set for my Intelligent Agents class at U Toronto Continuing Education.

lasimon

@lasimon

I also have Python 3.7 installed if that's a help (or a conflict).

lasimon

@lasimon

Tried it on my mac with and without setting the env variable and same result (libpython3.dylib image not found). Mac is also Python 3.6.

lasimon

@lasimon

Same thing on another mac running python 3.8

Shadaj Laddad

@shadaj

@lasimon oh, there's actually a big bug in the docs! You should be setting javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" (--prefix, instead of --ldflags). I'll update the docs to fix this shortly

lasimon

@lasimon

@shadaj Hmm. Still not working. config --prefix returns /usr. The .so is in /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/. The directory has two files in it: libpython3.7m.so libpython3.7.so

Shadaj Laddad

@shadaj

hmm, it seems like the python3-config is pointing to a system installation in /usr/lib instead of your python3.7 installation. Do you have a python3.7-config command available?

lasimon

@lasimon

I was able to install it using "sudo apt install python3.7-dev". If I now do "python3.7-config --prefix" it still replies with /usr.

Brandon Stilson

@bbstilson

Hey folks. I'm having trouble running the hello world, likely because my python environment is basically a superfund site.

scalapy-test:>python3-config --ldflags
-L/Users/brandons/.pyenv/versions/3.8.5/lib/python3.8/config-3.8-darwin -lintl -ldl -framework CoreFoundation

When I run the getting started example, i get the following stack trace:

sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>

this is my build.sbt

ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"

Brandon Stilson

@bbstilson

actually I made a docker container and ran into the same problem

https://gist.github.com/bbstilson/82ed6fd4826ac5a550455e7a8fd7aa6c

Shadaj Laddad

@shadaj

@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.

Brandon Stilson

@bbstilson

I got the same error sadly. I'll open an issue.

Travis

@travissarles

I'm getting the same issue as @bbstilson

Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0

Jakub Chrobasik

@jachro

We solved that issue by specifying:

VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
I hope that helps someone :)

1 reply

Peter Jansen

@peterjansen_ai_twitter

I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125

Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)

Peter Jansen

@peterjansen_ai_twitter

Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?

Peter Jansen

@peterjansen_ai_twitter

I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):

Even with the same array content

Peter Jansen

@peterjansen_ai_twitter

Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?

Pedro Luis

@PedroCorreiaLuis

I am struggling a lot, can someone help me what should I put in the sbt, thanks!

Shadaj Laddad

@shadaj

@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?

@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now

Kamil Kloch

@kamilkloch

Thanks for fixing python library fallback issues. Any chance for a bugfix release? :)

Shadaj Laddad

@shadaj

@kamilkloch v0.4.1 should be out very soon!

Kamil Kloch

@kamilkloch

@shadaj Thanks! I take this opportunity to thank you for all your work within the Scala community. Slinky, ScalaPy, who knows what brings 2021 ;) U da man.

Shadaj Laddad

@shadaj

@kamilkloch thanks so much, that means a lot! it's been very exciting to see the projects being helpful to the community!

Kamil Kloch

@kamilkloch

Hello all, the folowing snippet fails to compile:

import me.shadaj.scalapy.numpy.NumPy
import me.shadaj.scalapy.py

val np = py.module("numpy").as[NumPy]

could not find implicit value for evidence parameter of type me.shadaj.scalapy.readwrite.Reader[me.shadaj.scalapy.numpy.NumPy]
  val np = py.module("numpy").as[NumPy]

Any hints?

Also, py.module("numpy") compiles despite missing implicit jep: Jep, whereas implicitly[Jep] fails to compile.

Kamil Kloch

@kamilkloch

Update: it looks that the impiict jep instance is required by the (legacy?) artifact https://mvnrepository.com/artifact/me.shadaj/scalapy_2.12/0.2.0 sucked in with the scalapy-numpy facade.

Shadaj Laddad

@shadaj

@kamilkloch ah, yeah, I need to re-release those bindings for the new ScalaPy release

Anatolii Kmetiuk

@anatoliykmetyuk

Hi! I was following the Getting Started https://scalapy.dev/docs/ instructions, but when I tried running the code example, I got:

[error] Exception in thread "main" java.lang.ExceptionInInitializerError
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]     at Main$package$.hello(Main.scala:5)
[error]     at hello.main(Main.scala:4)
[error] Caused by: java.lang.Exception: Unable to locate Python library, tried python3, python3.7, python3.7m
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:19)
[error]     at scala.Option.getOrElse(Option.scala:201)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:19)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:102)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:102)
[error]     ... 4 more

Python was installed via homebrew, the OS is MacOS Catalina. The output of python3-config --configdir is /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin. When I run ls -1 $(python3-config --configdir), I get:

Makefile
Setup
Setup.local
config.c
config.c.in
install-sh
libpython3.9.a
libpython3.9.dylib
makesetup
python-config.py
python.o

So probably javaOptions += s"-Djava.library.path=${"python3-config --configdir".!!.trim}/lib" in the SBT settings cannot locate lib. How do I fix this?

Shadaj Laddad

@shadaj

@anatoliykmetyuk try adding SCALAPY_PYTHON_LIBRARY=python3.9 to your build to override the python version. I'll be publishing an update to the docs with guidance on this situation in just a bit!

Anatolii Kmetiuk

@anatoliykmetyuk

I tried doing so, this did not solve the problem but the error message got changed to include that version of python:

Caused by: java.lang.Exception: Unable to locate Python library, tried python3.9, python3, python3.7, python3.7

implisci

@implisci

Pre-Newbie here (as in not even used ScalaPy). Apologies if my questions are incorrect or assume wrong things. 1) Does ScalaPy load the PY interpreter(s) in an external process(es) and pass python code to them and await results? 2) The difference between something like Tensorflow for Scala and ScalaPy is that the former targets one library but this is (in principle) general? Thanks. Tensorflow if I recall might be in C/C++ with Python API so the second question may not be correct. Thanks.

Shadaj Laddad

@shadaj

@anatoliykmetyuk try python3.9m? at least on my macOS system that one works

@implisci yeah, that's generally the idea. ScalaPy actually integrates with the Python C APIs, though, so it can avoid generating Python code and have much faster interop. And 2 is exactly right, the goal is to be able to use existing Python libraries that often have a lot of logic baked in which is hard to port to Scala

Where communities thrive

People

Repo info

Activity