shadaj/scalapy

jhegedus42

@jhegedus42

Jozsefs-MacBook-Pro:scalapytest joco$ sbt
[info] Loading global plugins from /Users/joco/.sbt/1.0/plugins
[info] Loading project definition from /Users/joco/dev/ml/scalapy/test01/github/scalapytest/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to ScalaPyTest (in build file:/Users/joco/dev/ml/scalapy/test01/github/scalapytest/)
[info] sbt server started at local:///Users/joco/.sbt/1.0/server/3f6c4d00bf97ae0f90ba/sock
sbt:ScalaPyTest> run
[info] running (fork) example.Hello
[info] code runs
[error] Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initialize(MainInterpreter.java:183)
[error]     at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:97)
[error]     at jep.Jep.<init>(Jep.java:232)
[error]     at jep.Jep.<init>(Jep.java:228)
[error]     at me.shadaj.scalapy.py.JepInterpreter.<init>(JepInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.Platform$.newInterpreter(Platform.scala:5)
[error]     at me.shadaj.scalapy.py.package$.interpreter(package.scala:11)
[error]     at me.shadaj.scalapy.py.Writer$$anon$11.write(Writer.scala:75)
[error]     at me.shadaj.scalapy.py.Any$.from(Any.scala:28)
[error]     at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:6)
[error]     at example.Hello$delayedInit$body.apply(Hello.scala:3)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.Hello$.main(Hello.scala:3)
[error]     at example.Hello.main(Hello.scala)
[error] Caused by: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initializePython(Native Method)
[error]     at jep.MainInterpreter.access$100(MainInterpreter.java:64)
[error]     at jep.MainInterpreter$1.run(MainInterpreter.java:140)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Dec 24, 2019 11:51:02 PM
sbt:ScalaPyTest>

Eric K Richardson

@ekrich

Seems like something is wrong with the python install or something.

Maxime Kjaer

@MaximeKjaer

@jhegedus42 what version of Jep have you installed? You can try again with pip install jep==3.8.2

Eric K Richardson

@ekrich

I haven't tried honestly, but there is some discussion above that could help.

Shadaj Laddad

@shadaj

looks like a version mismatch as @MaximeKjaer noted, since some shared library is being loaded but there's a JNI definition that's missing

jhegedus42

@jhegedus42

thanks !

nice

this works

with 3.8.2 jep

thanks

how about jupyter ?

Maxime Kjaer

@MaximeKjaer

@jhegedus42 This looks like Ammonite doesn't fetch from the right resolver? I can't see anything else than 0.2.0 on Maven https://search.maven.org/artifact/me.shadaj/scalapy_2.12.

Ah, it could just be that you should be loading scalapy-core:0.3.0, which is on Maven https://search.maven.org/artifact/me.shadaj/scalapy-core_2.12

Btw, @shadaj, I think deployments have been broken since this build https://travis-ci.com/shadaj/scalapy/builds/142068062. It complains about:

[error] Multiple repositories are found:
[error] [meshadaj-1361] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1362] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1363] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1364] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1365] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1366] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1367] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1368] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1369] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1370] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1371] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1372] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1373] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] Specify one of the repository ids in the command line
[error] java.lang.IllegalStateException: Found multiple staging repositories
[error] Use 'last' for the full log.

Maxime Kjaer

@MaximeKjaer

This means that nothing has been published since 0.3.0+2. I'd like to submit a PR adding cross-building to Scala 2.13 to scalapy-numpy, but this is a blocker for that.

Maxime Kjaer

@MaximeKjaer

I have very little experience or knowledge of how publishing Scala projects works, but from what I can gather it seems (from this issue xerial/sbt-sonatype#78) that updating sbt-sonatype could help

jhegedus42

@jhegedus42

@MaximeKjaer hmmm, thanks, i guess for the time being I will be just playing around in colab... google, that is pretty neat ... and easy ...

Shadaj Laddad

@shadaj

oh thanks @MaximeKjaer for catching that, I'll clear out the Sonatype staging and re-publish for now

Maxime Kjaer

@MaximeKjaer

@shadaj Could you perhaps publish ScalaPy manually for now? I have scalapy-numpy cross building to 2.13 in this commit MaximeKjaer/scalapy-numpy@d8805dc but would need version 0.3.0+13-0c255e53 of ScalaPy on Maven in order to make builds reproducible

Shadaj Laddad

@shadaj

@MaximeKjaer cleared out staging repositories and restarted the build, it should be out soon!

Shadaj Laddad

@shadaj

I think I identified the source of the publish failures, just added a new commit that's publishing now

Shadaj Laddad

@shadaj

0.3.0+15-598682f0 has been released!

Maxime Kjaer

@MaximeKjaer

That's fantastic, thank you so much! I'm submitting a PR to scalapy-numpy now :)

Rohan Sircar

@rohan-sircar

Hi, does scalapy work with anaconda environments?

Shadaj Laddad

@shadaj

@nova999 yep, Anaconda correctly overrides the python3-config utility so as long as you use the javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" config it should pick up the current environment's Python

Rohan Sircar

@rohan-sircar

Thank you :)

Rohan Sircar

@rohan-sircar

So I got a simple pyopencl program working in scalapy. With typings it could be a good alternative to the seemingly abandoned ScalaCL.

Jorge

@jvican

nice! consider working on those typings then, i'm sure they'd be well received ;)

Rohan Sircar

@rohan-sircar

I did intend to. I was studying the numpy facade. Some things are still not quite clear to me. But I suppose I could lay a groundwork to begin with.

Maxime Kjaer

@MaximeKjaer

Based on previous discussion in this channel, I said I'd ping both @shadaj and @ekrich once I open sourced the shape-safe TensorFlow library for Dotty. So with a little bit of a delay, here's a link! https://github.com/MaximeKjaer/tf-dotty

Maxime Kjaer

@MaximeKjaer

@shadaj I copied the code of scalapy-tensorflow into a scalapy-tensorflow module. This is just the easiest way that I found to augment the existing typings while working on tf-dotty. I would be happy to contribute them back to the main scalapy-tensorflow project, once I've typed a full TF module

Shadaj Laddad

@shadaj

@MaximeKjaer this is very exciting! Took a quick glance over the website just now, looks very nice! Also was cool to see the thought process that went into the implementation

Mark Melton

@melton1968

I'm having trouble getting Intellij to work correctly for the cross build. It is very grumpy about CPythonAPI and Platform when I am visiting any files in the shared sources. Anyone else have / solved this issue?

Shadaj Laddad

@shadaj

@melton1968 one strategy could be to just exclude either the JVM or Native sources so that IntelliJ doesn't get confused about the duplicate definitions. I've been using Metals with VS Code these days, which works pretty well for cross builds, though it doesn't have as many of the refactoring abilities as IntelliJ yet.

Piotr Gawryś

@Avasil

Which jep version should be used for 0.3.0+17-2bfe86de ? I tried both 3.8.2 and 3.9.0 and both result in following error:

[error] Native library (darwin/libpython3.7m.dylib) not found in resource path (/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/job-2/target/90ed79f3/562ee7c8/scala-python-example_2.12-1.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/567bf6c2/9e2a823f/jep-3.9.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/bc8bc145/464fe2e1/scala-library.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/aa0ecde6/28d0208b/scalapy-macros_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/1a583173/abb78bc4/jna-5.4.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/26ba9c6e/34d55877/scalapy-core_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/2ce76994/5a36165d/scala-reflect.jar)
[error]  at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
  [error]  at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
  [error]  at com.sun.jna.Native.register(Native.java:1722)
  [error]  at com.sun.jna.Native.register(Native.java:1443)
  [error]  at me.shadaj.scalapy.py.CPythonAPIInterface.<init>(CPythonAPI.scala:6)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<init>(CPythonAPI.scala:74)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<clinit>(CPythonAPI.scala)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<init>(CPythonInterpreter.scala:4)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala)
  [error]  at me.shadaj.scalapy.py.package$.<init>(package.scala:12)
[error]  at me.shadaj.scalapy.py.package$.<clinit>(package.scala)
[error]  at com.sushant.example.PiotrekExample$.delayedEndpoint$com$sushant$example$PiotrekExample$1(PiotrekExample.scala:13)
  [error]  at com.sushant.example.PiotrekExample$delayedInit$body.apply(PiotrekExample.scala:8)
  [error]  at scala.Function0.apply$mcV$sp(Function0.scala:39)
  [error]  at scala.Function0.apply$mcV$sp$(Function0.scala:39)
  [error]  at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
  [error]  at scala.App.$anonfun$main$1$adapted(App.scala:80)
  [error]  at scala.collection.immutable.List.foreach(List.scala:392)
  [error]  at scala.App.main(App.scala:80)
  [error]  at scala.App.main$(App.scala:78)
  [error]  at com.sushant.example.PiotrekExample$.main(PiotrekExample.scala:8)
  [error]  at com.sushant.example.PiotrekExample.main(PiotrekExample.scala)
  [error]  Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.7m.dylib, 9): image not found
  [error]   at com.sun.jna.Native.open(Native Method)
  [error]   at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
  [error]   ... 21 more

The same code works with scalapy 0.3.0. Any idea what could be missing from my setup?
It fails on val tf = py.module("tensorflow")

Shadaj Laddad

@shadaj

@Avasil the new ScalaPy release doesn't need Jep anymore, it uses your existing Python installation directly. You can add your installation to the path with javaOptions in Test += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" in your SBT settings

Piotr Gawryś

@Avasil

Thank you, it's working fine now!

Piotr Gawryś

@Avasil

Is there any example of a class which can be created by the user calling constructor directly? e.g.

@py.native class Conv2D(
                       filters: Int,
                       kernel_size: py.|[Int, (Int, Int)],
                       strides: py.|[Int, (Int, Int)] = (1, 1),
                       padding: String = "valid",
                       data_format: String = "channels_last",
                       dilation_rate: py.|[Int, (Int, Int)] = (1, 1),
                       activation: Option[String] = None,
                       use_bias: Boolean = true,
                       kernel_initializer: String = "glorot_uniform",
                       bias_initializer: String = "zeros",
                       kernel_regularizer: Option[String] = None,
                       bias_regularizer: Option[String] = None,
                       activity_regularizer: Option[String] = None,
                       kernel_constraint: Option[String] = None,
                       bias_constraint: Option[String] = None
                       ) extends py.Object

I would like to be able to call it like Conv2D(32, kernel_size = (3, 3)) but I'm not sure how to provide PyValue

Shadaj Laddad

@shadaj

@Avasil there's no class equivalent yet in ScalaPy (this is on my mind but will likely require fairly complex macro transformations), so you can just use objects where the apply method calls a method facade

Piotr Gawryś

@Avasil

@shadaj Thanks, I will try it. Do you mean something like:

@py.native abstract class MyClass(x: String) extends py.Object

@py.native object MyClass extends py.Object {
  def apply(x: String): MyClass = py.native
}

Although I need to provide Reader[MyClass] and PyValue for the object itself
I have some classes working if there is a single parameter (val value: PyValue) but I have trouble to do it if there are more

Shadaj Laddad

@shadaj

@Avasil MyClass would have to be a trait with x being defined as a method inside. Right now, ScalaPy also doesn't support top-level objects as facades, so you would need to define a separate trait which contains the constructor which gets a bit messy. What I'd suggest is defining the constructor inside the facade for the containing module, similar to how https://github.com/shadaj/scalapy-tensorflow/blob/master/src/main/scala/me/shadaj/scalapy/tensorflow/TensorFlow.scala#L52 defines the constructor for Variable.

This is not too elegant unfortunately, but improving this is my top priority after stabilizing the new interpreter backend so should improve soon!

lasimon

@lasimon

Trying version 0.4.0. Updated build.sbt to use the new javaOptions but now I'm getting: linux-x86-64/libpython3.so not found. I have a libpython3.6.so

lasimon

@lasimon

Also a 3.7.so

Shadaj Laddad

@shadaj

@lasimon hmm, that's odd; what's the exact error you're seeing? ScalaPy should try to fall back to other library names when initializing the interpreter

lasimon

@lasimon

@shadaj build.sbt:

fork := true
import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
// javaOptions += "-Djna.library.path=/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib"

[info] running (fork) wumpusWorld.DQNWorld
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] Native library (linux-x86-64/libpython3.so) not found in resource path ([file:/tmp/sbt_82d1e13c/job-1/target/55c69968/832b95f1/wumpusworld_2.12-0.1.jar, file:/tmp/sbt_82d1e13c/target/dc7f77ac/b41586e0/scala-library-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/6a07f7c0/c6806aaf/refined_2.12-0.9.15.jar, file:/tmp/sbt_82d1e13c/target/f631ecf1/e854b270/figaro_2.12-5.0.0.0.jar, file:/tmp/sbt_82d1e13c/target/da834eb9/baaae62c/scalapy-core_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/95d6cb8d/f93cbdeb/scala-compiler-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/17f40488/b6b93b90/scala-reflect-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/df126de0/b50eae97/shapeless_2.12-2.3.3.jar, file:/tmp/sbt_82d1e13c/target/906e7fb1/01c9646c/scala-xml_2.12-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/b81ef6b7/795641c2/asm-3.3.1.jar, file:/tmp/sbt_82d1e13c/target/a5cd33ac/662f0474/commons-math3-3.3.jar, file:/tmp/sbt_82d1e13c/target/0421973b/0603d5cb/jsci-1.2.jar, file:/tmp/sbt_82d1e13c/target/d88c303f/81fcec1e/akka-actor_2.12-2.4.18.jar, file:/tmp/sbt_82d1e13c/target/ea9de1d2/c7269f1b/breeze_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/29f545e8/0f3e3243/argonaut_2.12-6.2.jar, file:/tmp/sbt_82d1e13c/target/846ef561/db21d5e7/prefuse-beta-20071021.jar, file:/tmp/sbt_82d1e13c/target/7bc43030/382b5f00/scala-swing_2.12-2.0.0.jar, file:/tmp/sbt_82d1e13c/target/8af72ca7/8c48d6d7/scalapy-macros_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/2b3b90b1/f37928f3/jna-5.5.0.jar, file:/tmp/sbt_82d1e13c/target/f7ef733c/1d4f9fbf/macro-compat_2.12-1.1.1.jar, file:/tmp/sbt_82d1e13c/target/496371c0/d0443e0c/lpsolve-2003-11-03.jar, file:/tmp/sbt_82d1e13c/target/7546bb71/2faab0fb/mathml-dom-java-2.0.jar, file:/tmp/sbt_82d1e13c/target/4399f739/4f47a0d5/xercesImpl-2.8.1.jar, file:/tmp/sbt_82d1e13c/target/c5d711c7/d61d4b86/config-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/f679673a/b69204cf/scala-java8-compat_2.12-0.8.0.jar, file:/tmp/sbt_82d1e13c/target/84087de9/b5315cdf/breeze-macros_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/419dc0c7/453aa781/core-1.1.2.jar, file:/tmp/sbt_82d1e13c/target/2ac29a43/764709f2/arpack_combined_all-0.1.jar, file:/tmp/sbt_82d1e13c/target/47c97988/b010f01b/opencsv-2.3.jar, file:/tmp/sbt_82d1e13c/target/cfb0976e/6518a453/jtransforms-2.4.0.jar, file:/tmp/sbt_82d1e13c/target/e10d518a/14229e37/spire_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/cb619cd6/dd21cb48/slf4j-api-1.7.5.jar, file:/tmp/sbt_82d1e13c/target/4a189398/a8781ea4/xml-apis-1.3.03.jar, file:/tmp/sbt_82d1e13c/target/f8a5c36f/ff1b5461/junit-4.8.2.jar, file:/tmp/sbt_82d1e13c/target/84ed269c/8778977b/spire-macros_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/b67dc7de/2334303a/machinist_2.12-0.6.1.jar])
[error] at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error] at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error] at com.sun.jna.Native.register(Native.java:1722)
[error] at com.sun.jna.Native.register(Native.java:1443)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] at scala.util.Try$.apply(Try.scala:213)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error] at scala.collection.LinearSeqOptimized.find(LinearSeqOptimized.scala:115)
[error] at scala.collection.LinearSeqOptimized.find$(LinearSeqOptimized.scala:112)
[error] at scala.collection.immutable.List.find(List.scala:91)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95

python3-config --ldflags
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

Where communities thrive

People

Repo info

Activity