shadaj/scalapy

lasimon

@lasimon

Trying version 0.4.0. Updated build.sbt to use the new javaOptions but now I'm getting: linux-x86-64/libpython3.so not found. I have a libpython3.6.so

lasimon

@lasimon

Also a 3.7.so

Shadaj Laddad

@shadaj

@lasimon hmm, that's odd; what's the exact error you're seeing? ScalaPy should try to fall back to other library names when initializing the interpreter

lasimon

@lasimon

@shadaj build.sbt:

fork := true
import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
// javaOptions += "-Djna.library.path=/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib"

[info] running (fork) wumpusWorld.DQNWorld
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] Native library (linux-x86-64/libpython3.so) not found in resource path ([file:/tmp/sbt_82d1e13c/job-1/target/55c69968/832b95f1/wumpusworld_2.12-0.1.jar, file:/tmp/sbt_82d1e13c/target/dc7f77ac/b41586e0/scala-library-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/6a07f7c0/c6806aaf/refined_2.12-0.9.15.jar, file:/tmp/sbt_82d1e13c/target/f631ecf1/e854b270/figaro_2.12-5.0.0.0.jar, file:/tmp/sbt_82d1e13c/target/da834eb9/baaae62c/scalapy-core_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/95d6cb8d/f93cbdeb/scala-compiler-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/17f40488/b6b93b90/scala-reflect-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/df126de0/b50eae97/shapeless_2.12-2.3.3.jar, file:/tmp/sbt_82d1e13c/target/906e7fb1/01c9646c/scala-xml_2.12-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/b81ef6b7/795641c2/asm-3.3.1.jar, file:/tmp/sbt_82d1e13c/target/a5cd33ac/662f0474/commons-math3-3.3.jar, file:/tmp/sbt_82d1e13c/target/0421973b/0603d5cb/jsci-1.2.jar, file:/tmp/sbt_82d1e13c/target/d88c303f/81fcec1e/akka-actor_2.12-2.4.18.jar, file:/tmp/sbt_82d1e13c/target/ea9de1d2/c7269f1b/breeze_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/29f545e8/0f3e3243/argonaut_2.12-6.2.jar, file:/tmp/sbt_82d1e13c/target/846ef561/db21d5e7/prefuse-beta-20071021.jar, file:/tmp/sbt_82d1e13c/target/7bc43030/382b5f00/scala-swing_2.12-2.0.0.jar, file:/tmp/sbt_82d1e13c/target/8af72ca7/8c48d6d7/scalapy-macros_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/2b3b90b1/f37928f3/jna-5.5.0.jar, file:/tmp/sbt_82d1e13c/target/f7ef733c/1d4f9fbf/macro-compat_2.12-1.1.1.jar, file:/tmp/sbt_82d1e13c/target/496371c0/d0443e0c/lpsolve-2003-11-03.jar, file:/tmp/sbt_82d1e13c/target/7546bb71/2faab0fb/mathml-dom-java-2.0.jar, file:/tmp/sbt_82d1e13c/target/4399f739/4f47a0d5/xercesImpl-2.8.1.jar, file:/tmp/sbt_82d1e13c/target/c5d711c7/d61d4b86/config-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/f679673a/b69204cf/scala-java8-compat_2.12-0.8.0.jar, file:/tmp/sbt_82d1e13c/target/84087de9/b5315cdf/breeze-macros_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/419dc0c7/453aa781/core-1.1.2.jar, file:/tmp/sbt_82d1e13c/target/2ac29a43/764709f2/arpack_combined_all-0.1.jar, file:/tmp/sbt_82d1e13c/target/47c97988/b010f01b/opencsv-2.3.jar, file:/tmp/sbt_82d1e13c/target/cfb0976e/6518a453/jtransforms-2.4.0.jar, file:/tmp/sbt_82d1e13c/target/e10d518a/14229e37/spire_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/cb619cd6/dd21cb48/slf4j-api-1.7.5.jar, file:/tmp/sbt_82d1e13c/target/4a189398/a8781ea4/xml-apis-1.3.03.jar, file:/tmp/sbt_82d1e13c/target/f8a5c36f/ff1b5461/junit-4.8.2.jar, file:/tmp/sbt_82d1e13c/target/84ed269c/8778977b/spire-macros_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/b67dc7de/2334303a/machinist_2.12-0.6.1.jar])
[error] at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error] at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error] at com.sun.jna.Native.register(Native.java:1722)
[error] at com.sun.jna.Native.register(Native.java:1443)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] at scala.util.Try$.apply(Try.scala:213)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error] at scala.collection.LinearSeqOptimized.find(LinearSeqOptimized.scala:115)
[error] at scala.collection.LinearSeqOptimized.find$(LinearSeqOptimized.scala:112)
[error] at scala.collection.immutable.List.find(List.scala:91)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95

python3-config --ldflags
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

lasimon

@lasimon

larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ cd /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ ls
config.c install-sh libpython3.6m-pic.a libpython3.6.so makesetup python.o Setup.config
config.c.in libpython3.6m.a libpython3.6m.so Makefile python-config.py Setup Setup.local

Shadaj Laddad

@shadaj

Ah, looks like you have Python 3.6, which ScalaPy doesn't load by default (though perhaps it should). For now, could you try adding the environment variable SCALAPY_PYTHON_LIBRARY=python3.6 before launching SBT?

lasimon

@lasimon

larrysimon@inspiron:~/IdeaProjects/wumpus$ echo $SCALAPY_PYTHON_LIBRARY
python3.6
then sbt run
... same error

BTW what I'm doing is a reference implementation of a solution to the classic Russell & Norvig Wumpus World using Deep RL on pyTorch. A Python version is an assignment I set for my Intelligent Agents class at U Toronto Continuing Education.

lasimon

@lasimon

I also have Python 3.7 installed if that's a help (or a conflict).

lasimon

@lasimon

Tried it on my mac with and without setting the env variable and same result (libpython3.dylib image not found). Mac is also Python 3.6.

lasimon

@lasimon

Same thing on another mac running python 3.8

Shadaj Laddad

@shadaj

@lasimon oh, there's actually a big bug in the docs! You should be setting javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" (--prefix, instead of --ldflags). I'll update the docs to fix this shortly

lasimon

@lasimon

@shadaj Hmm. Still not working. config --prefix returns /usr. The .so is in /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/. The directory has two files in it: libpython3.7m.so libpython3.7.so

Shadaj Laddad

@shadaj

hmm, it seems like the python3-config is pointing to a system installation in /usr/lib instead of your python3.7 installation. Do you have a python3.7-config command available?

lasimon

@lasimon

I was able to install it using "sudo apt install python3.7-dev". If I now do "python3.7-config --prefix" it still replies with /usr.

Brandon Stilson

@bbstilson

Hey folks. I'm having trouble running the hello world, likely because my python environment is basically a superfund site.

scalapy-test:>python3-config --ldflags
-L/Users/brandons/.pyenv/versions/3.8.5/lib/python3.8/config-3.8-darwin -lintl -ldl -framework CoreFoundation

When I run the getting started example, i get the following stack trace:

sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>

this is my build.sbt

ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"

Brandon Stilson

@bbstilson

actually I made a docker container and ran into the same problem

https://gist.github.com/bbstilson/82ed6fd4826ac5a550455e7a8fd7aa6c

Shadaj Laddad

@shadaj

@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.

Brandon Stilson

@bbstilson

I got the same error sadly. I'll open an issue.

Travis

@travissarles

I'm getting the same issue as @bbstilson

Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0

Jakub Chrobasik

@jachro

We solved that issue by specifying:

VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
I hope that helps someone :)

1 reply

Peter Jansen

@peterjansen_ai_twitter

I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125

Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)

Peter Jansen

@peterjansen_ai_twitter

Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?

Peter Jansen

@peterjansen_ai_twitter

I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):

Even with the same array content

Peter Jansen

@peterjansen_ai_twitter

Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?

Pedro Luis

@PedroCorreiaLuis

I am struggling a lot, can someone help me what should I put in the sbt, thanks!

Shadaj Laddad

@shadaj

@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?

@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now

Kamil Kloch

@kamilkloch

Thanks for fixing python library fallback issues. Any chance for a bugfix release? :)

Shadaj Laddad

@shadaj

@kamilkloch v0.4.1 should be out very soon!

Kamil Kloch

@kamilkloch

@shadaj Thanks! I take this opportunity to thank you for all your work within the Scala community. Slinky, ScalaPy, who knows what brings 2021 ;) U da man.

Shadaj Laddad

@shadaj

@kamilkloch thanks so much, that means a lot! it's been very exciting to see the projects being helpful to the community!

Kamil Kloch

@kamilkloch

Hello all, the folowing snippet fails to compile:

import me.shadaj.scalapy.numpy.NumPy
import me.shadaj.scalapy.py

val np = py.module("numpy").as[NumPy]

could not find implicit value for evidence parameter of type me.shadaj.scalapy.readwrite.Reader[me.shadaj.scalapy.numpy.NumPy]
  val np = py.module("numpy").as[NumPy]

Any hints?

Also, py.module("numpy") compiles despite missing implicit jep: Jep, whereas implicitly[Jep] fails to compile.

Kamil Kloch

@kamilkloch

Update: it looks that the impiict jep instance is required by the (legacy?) artifact https://mvnrepository.com/artifact/me.shadaj/scalapy_2.12/0.2.0 sucked in with the scalapy-numpy facade.

Shadaj Laddad

@shadaj

@kamilkloch ah, yeah, I need to re-release those bindings for the new ScalaPy release

Where communities thrive

People

Repo info

Activity