shadaj/scalapy

Shadaj Laddad

@shadaj

@melton1968 one strategy could be to just exclude either the JVM or Native sources so that IntelliJ doesn't get confused about the duplicate definitions. I've been using Metals with VS Code these days, which works pretty well for cross builds, though it doesn't have as many of the refactoring abilities as IntelliJ yet.

Piotr Gawryś

@Avasil

Which jep version should be used for 0.3.0+17-2bfe86de ? I tried both 3.8.2 and 3.9.0 and both result in following error:

[error] Native library (darwin/libpython3.7m.dylib) not found in resource path (/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/job-2/target/90ed79f3/562ee7c8/scala-python-example_2.12-1.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/567bf6c2/9e2a823f/jep-3.9.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/bc8bc145/464fe2e1/scala-library.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/aa0ecde6/28d0208b/scalapy-macros_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/1a583173/abb78bc4/jna-5.4.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/26ba9c6e/34d55877/scalapy-core_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/2ce76994/5a36165d/scala-reflect.jar)
[error]  at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
  [error]  at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
  [error]  at com.sun.jna.Native.register(Native.java:1722)
  [error]  at com.sun.jna.Native.register(Native.java:1443)
  [error]  at me.shadaj.scalapy.py.CPythonAPIInterface.<init>(CPythonAPI.scala:6)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<init>(CPythonAPI.scala:74)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<clinit>(CPythonAPI.scala)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<init>(CPythonInterpreter.scala:4)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala)
  [error]  at me.shadaj.scalapy.py.package$.<init>(package.scala:12)
[error]  at me.shadaj.scalapy.py.package$.<clinit>(package.scala)
[error]  at com.sushant.example.PiotrekExample$.delayedEndpoint$com$sushant$example$PiotrekExample$1(PiotrekExample.scala:13)
  [error]  at com.sushant.example.PiotrekExample$delayedInit$body.apply(PiotrekExample.scala:8)
  [error]  at scala.Function0.apply$mcV$sp(Function0.scala:39)
  [error]  at scala.Function0.apply$mcV$sp$(Function0.scala:39)
  [error]  at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
  [error]  at scala.App.$anonfun$main$1$adapted(App.scala:80)
  [error]  at scala.collection.immutable.List.foreach(List.scala:392)
  [error]  at scala.App.main(App.scala:80)
  [error]  at scala.App.main$(App.scala:78)
  [error]  at com.sushant.example.PiotrekExample$.main(PiotrekExample.scala:8)
  [error]  at com.sushant.example.PiotrekExample.main(PiotrekExample.scala)
  [error]  Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.7m.dylib, 9): image not found
  [error]   at com.sun.jna.Native.open(Native Method)
  [error]   at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
  [error]   ... 21 more

The same code works with scalapy 0.3.0. Any idea what could be missing from my setup?
It fails on val tf = py.module("tensorflow")

Shadaj Laddad

@shadaj

@Avasil the new ScalaPy release doesn't need Jep anymore, it uses your existing Python installation directly. You can add your installation to the path with javaOptions in Test += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" in your SBT settings

Piotr Gawryś

@Avasil

Thank you, it's working fine now!

Piotr Gawryś

@Avasil

Is there any example of a class which can be created by the user calling constructor directly? e.g.

@py.native class Conv2D(
                       filters: Int,
                       kernel_size: py.|[Int, (Int, Int)],
                       strides: py.|[Int, (Int, Int)] = (1, 1),
                       padding: String = "valid",
                       data_format: String = "channels_last",
                       dilation_rate: py.|[Int, (Int, Int)] = (1, 1),
                       activation: Option[String] = None,
                       use_bias: Boolean = true,
                       kernel_initializer: String = "glorot_uniform",
                       bias_initializer: String = "zeros",
                       kernel_regularizer: Option[String] = None,
                       bias_regularizer: Option[String] = None,
                       activity_regularizer: Option[String] = None,
                       kernel_constraint: Option[String] = None,
                       bias_constraint: Option[String] = None
                       ) extends py.Object

I would like to be able to call it like Conv2D(32, kernel_size = (3, 3)) but I'm not sure how to provide PyValue

Shadaj Laddad

@shadaj

@Avasil there's no class equivalent yet in ScalaPy (this is on my mind but will likely require fairly complex macro transformations), so you can just use objects where the apply method calls a method facade

Piotr Gawryś

@Avasil

@shadaj Thanks, I will try it. Do you mean something like:

@py.native abstract class MyClass(x: String) extends py.Object

@py.native object MyClass extends py.Object {
  def apply(x: String): MyClass = py.native
}

Although I need to provide Reader[MyClass] and PyValue for the object itself
I have some classes working if there is a single parameter (val value: PyValue) but I have trouble to do it if there are more

Shadaj Laddad

@shadaj

@Avasil MyClass would have to be a trait with x being defined as a method inside. Right now, ScalaPy also doesn't support top-level objects as facades, so you would need to define a separate trait which contains the constructor which gets a bit messy. What I'd suggest is defining the constructor inside the facade for the containing module, similar to how https://github.com/shadaj/scalapy-tensorflow/blob/master/src/main/scala/me/shadaj/scalapy/tensorflow/TensorFlow.scala#L52 defines the constructor for Variable.

This is not too elegant unfortunately, but improving this is my top priority after stabilizing the new interpreter backend so should improve soon!

lasimon

@lasimon

Trying version 0.4.0. Updated build.sbt to use the new javaOptions but now I'm getting: linux-x86-64/libpython3.so not found. I have a libpython3.6.so

lasimon

@lasimon

Also a 3.7.so

Shadaj Laddad

@shadaj

@lasimon hmm, that's odd; what's the exact error you're seeing? ScalaPy should try to fall back to other library names when initializing the interpreter

lasimon

@lasimon

@shadaj build.sbt:

fork := true
import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
// javaOptions += "-Djna.library.path=/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib"

[info] running (fork) wumpusWorld.DQNWorld
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] Native library (linux-x86-64/libpython3.so) not found in resource path ([file:/tmp/sbt_82d1e13c/job-1/target/55c69968/832b95f1/wumpusworld_2.12-0.1.jar, file:/tmp/sbt_82d1e13c/target/dc7f77ac/b41586e0/scala-library-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/6a07f7c0/c6806aaf/refined_2.12-0.9.15.jar, file:/tmp/sbt_82d1e13c/target/f631ecf1/e854b270/figaro_2.12-5.0.0.0.jar, file:/tmp/sbt_82d1e13c/target/da834eb9/baaae62c/scalapy-core_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/95d6cb8d/f93cbdeb/scala-compiler-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/17f40488/b6b93b90/scala-reflect-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/df126de0/b50eae97/shapeless_2.12-2.3.3.jar, file:/tmp/sbt_82d1e13c/target/906e7fb1/01c9646c/scala-xml_2.12-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/b81ef6b7/795641c2/asm-3.3.1.jar, file:/tmp/sbt_82d1e13c/target/a5cd33ac/662f0474/commons-math3-3.3.jar, file:/tmp/sbt_82d1e13c/target/0421973b/0603d5cb/jsci-1.2.jar, file:/tmp/sbt_82d1e13c/target/d88c303f/81fcec1e/akka-actor_2.12-2.4.18.jar, file:/tmp/sbt_82d1e13c/target/ea9de1d2/c7269f1b/breeze_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/29f545e8/0f3e3243/argonaut_2.12-6.2.jar, file:/tmp/sbt_82d1e13c/target/846ef561/db21d5e7/prefuse-beta-20071021.jar, file:/tmp/sbt_82d1e13c/target/7bc43030/382b5f00/scala-swing_2.12-2.0.0.jar, file:/tmp/sbt_82d1e13c/target/8af72ca7/8c48d6d7/scalapy-macros_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/2b3b90b1/f37928f3/jna-5.5.0.jar, file:/tmp/sbt_82d1e13c/target/f7ef733c/1d4f9fbf/macro-compat_2.12-1.1.1.jar, file:/tmp/sbt_82d1e13c/target/496371c0/d0443e0c/lpsolve-2003-11-03.jar, file:/tmp/sbt_82d1e13c/target/7546bb71/2faab0fb/mathml-dom-java-2.0.jar, file:/tmp/sbt_82d1e13c/target/4399f739/4f47a0d5/xercesImpl-2.8.1.jar, file:/tmp/sbt_82d1e13c/target/c5d711c7/d61d4b86/config-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/f679673a/b69204cf/scala-java8-compat_2.12-0.8.0.jar, file:/tmp/sbt_82d1e13c/target/84087de9/b5315cdf/breeze-macros_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/419dc0c7/453aa781/core-1.1.2.jar, file:/tmp/sbt_82d1e13c/target/2ac29a43/764709f2/arpack_combined_all-0.1.jar, file:/tmp/sbt_82d1e13c/target/47c97988/b010f01b/opencsv-2.3.jar, file:/tmp/sbt_82d1e13c/target/cfb0976e/6518a453/jtransforms-2.4.0.jar, file:/tmp/sbt_82d1e13c/target/e10d518a/14229e37/spire_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/cb619cd6/dd21cb48/slf4j-api-1.7.5.jar, file:/tmp/sbt_82d1e13c/target/4a189398/a8781ea4/xml-apis-1.3.03.jar, file:/tmp/sbt_82d1e13c/target/f8a5c36f/ff1b5461/junit-4.8.2.jar, file:/tmp/sbt_82d1e13c/target/84ed269c/8778977b/spire-macros_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/b67dc7de/2334303a/machinist_2.12-0.6.1.jar])
[error] at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error] at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error] at com.sun.jna.Native.register(Native.java:1722)
[error] at com.sun.jna.Native.register(Native.java:1443)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] at scala.util.Try$.apply(Try.scala:213)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error] at scala.collection.LinearSeqOptimized.find(LinearSeqOptimized.scala:115)
[error] at scala.collection.LinearSeqOptimized.find$(LinearSeqOptimized.scala:112)
[error] at scala.collection.immutable.List.find(List.scala:91)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95

python3-config --ldflags
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

lasimon

@lasimon

larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ cd /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ ls
config.c install-sh libpython3.6m-pic.a libpython3.6.so makesetup python.o Setup.config
config.c.in libpython3.6m.a libpython3.6m.so Makefile python-config.py Setup Setup.local

Shadaj Laddad

@shadaj

Ah, looks like you have Python 3.6, which ScalaPy doesn't load by default (though perhaps it should). For now, could you try adding the environment variable SCALAPY_PYTHON_LIBRARY=python3.6 before launching SBT?

lasimon

@lasimon

larrysimon@inspiron:~/IdeaProjects/wumpus$ echo $SCALAPY_PYTHON_LIBRARY
python3.6
then sbt run
... same error

BTW what I'm doing is a reference implementation of a solution to the classic Russell & Norvig Wumpus World using Deep RL on pyTorch. A Python version is an assignment I set for my Intelligent Agents class at U Toronto Continuing Education.

lasimon

@lasimon

I also have Python 3.7 installed if that's a help (or a conflict).

lasimon

@lasimon

Tried it on my mac with and without setting the env variable and same result (libpython3.dylib image not found). Mac is also Python 3.6.

lasimon

@lasimon

Same thing on another mac running python 3.8

Shadaj Laddad

@shadaj

@lasimon oh, there's actually a big bug in the docs! You should be setting javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" (--prefix, instead of --ldflags). I'll update the docs to fix this shortly

lasimon

@lasimon

@shadaj Hmm. Still not working. config --prefix returns /usr. The .so is in /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/. The directory has two files in it: libpython3.7m.so libpython3.7.so

Shadaj Laddad

@shadaj

hmm, it seems like the python3-config is pointing to a system installation in /usr/lib instead of your python3.7 installation. Do you have a python3.7-config command available?

lasimon

@lasimon

I was able to install it using "sudo apt install python3.7-dev". If I now do "python3.7-config --prefix" it still replies with /usr.

Brandon Stilson

@bbstilson

Hey folks. I'm having trouble running the hello world, likely because my python environment is basically a superfund site.

scalapy-test:>python3-config --ldflags
-L/Users/brandons/.pyenv/versions/3.8.5/lib/python3.8/config-3.8-darwin -lintl -ldl -framework CoreFoundation

When I run the getting started example, i get the following stack trace:

sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>

this is my build.sbt

ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"

Brandon Stilson

@bbstilson

actually I made a docker container and ran into the same problem

https://gist.github.com/bbstilson/82ed6fd4826ac5a550455e7a8fd7aa6c

Shadaj Laddad

@shadaj

@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.

Brandon Stilson

@bbstilson

I got the same error sadly. I'll open an issue.

Travis

@travissarles

I'm getting the same issue as @bbstilson

Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0

Jakub Chrobasik

@jachro

We solved that issue by specifying:

VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
I hope that helps someone :)

1 reply

Peter Jansen

@peterjansen_ai_twitter

I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125

Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)

Peter Jansen

@peterjansen_ai_twitter

Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?

Peter Jansen

@peterjansen_ai_twitter

I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):

Even with the same array content

Peter Jansen

@peterjansen_ai_twitter

Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?

Pedro Luis

@PedroCorreiaLuis

I am struggling a lot, can someone help me what should I put in the sbt, thanks!

Shadaj Laddad

@shadaj

@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?

@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now

Where communities thrive

People

Repo info

Activity