Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 23 06:23
    anatoliykmetyuk commented #195
  • Sep 23 00:53
    scala-steward synchronize #222
  • Sep 23 00:53
    scala-steward opened #224
  • Sep 23 00:53
    scala-steward synchronize #184
  • Sep 23 00:53
    scala-steward synchronize #211
  • Sep 22 21:28

    shadaj on main

    Migration to Scala 3 (#195) 1.… (compare)

  • Sep 22 21:28
    shadaj closed #195
  • Sep 22 18:38
    Ang9876 commented #195
  • Sep 22 18:36
    Ang9876 synchronize #195
  • Sep 20 21:39

    dependabot[bot] on npm_and_yarn

    (compare)

  • Sep 20 21:39
    dependabot[bot] closed #213
  • Sep 20 21:39
    dependabot[bot] commented #213
  • Sep 20 21:39
    dependabot[bot] labeled #223
  • Sep 20 21:39
    dependabot[bot] opened #223
  • Sep 20 21:39

    dependabot[bot] on npm_and_yarn

    Bump prismjs from 1.23.0 to 1.2… (compare)

  • Sep 18 01:09
    scala-steward opened #222
  • Sep 15 15:43
    Ang9876 commented #195
  • Sep 15 15:42
    Ang9876 synchronize #195
  • Sep 15 00:26
    scala-steward closed #185
  • Sep 15 00:26
    scala-steward commented #185
Shadaj Laddad
@shadaj
@melton1968 one strategy could be to just exclude either the JVM or Native sources so that IntelliJ doesn't get confused about the duplicate definitions. I've been using Metals with VS Code these days, which works pretty well for cross builds, though it doesn't have as many of the refactoring abilities as IntelliJ yet.
Piotr Gawryś
@Avasil

Which jep version should be used for 0.3.0+17-2bfe86de ? I tried both 3.8.2 and 3.9.0 and both result in following error:

[error] Native library (darwin/libpython3.7m.dylib) not found in resource path (/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/job-2/target/90ed79f3/562ee7c8/scala-python-example_2.12-1.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/567bf6c2/9e2a823f/jep-3.9.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/bc8bc145/464fe2e1/scala-library.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/aa0ecde6/28d0208b/scalapy-macros_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/1a583173/abb78bc4/jna-5.4.0.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/26ba9c6e/34d55877/scalapy-core_2.12-0.3.0%2B17-2bfe86de.jar:/var/folders/lr/x_gb25v56fj3_80kyyk17lhc0000gn/T/sbt_dd5aa2c1/target/2ce76994/5a36165d/scala-reflect.jar)
[error]  at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
  [error]  at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
  [error]  at com.sun.jna.Native.register(Native.java:1722)
  [error]  at com.sun.jna.Native.register(Native.java:1443)
  [error]  at me.shadaj.scalapy.py.CPythonAPIInterface.<init>(CPythonAPI.scala:6)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<init>(CPythonAPI.scala:74)
  [error]  at me.shadaj.scalapy.py.CPythonAPI$.<clinit>(CPythonAPI.scala)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<init>(CPythonInterpreter.scala:4)
  [error]  at me.shadaj.scalapy.py.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala)
  [error]  at me.shadaj.scalapy.py.package$.<init>(package.scala:12)
[error]  at me.shadaj.scalapy.py.package$.<clinit>(package.scala)
[error]  at com.sushant.example.PiotrekExample$.delayedEndpoint$com$sushant$example$PiotrekExample$1(PiotrekExample.scala:13)
  [error]  at com.sushant.example.PiotrekExample$delayedInit$body.apply(PiotrekExample.scala:8)
  [error]  at scala.Function0.apply$mcV$sp(Function0.scala:39)
  [error]  at scala.Function0.apply$mcV$sp$(Function0.scala:39)
  [error]  at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
  [error]  at scala.App.$anonfun$main$1$adapted(App.scala:80)
  [error]  at scala.collection.immutable.List.foreach(List.scala:392)
  [error]  at scala.App.main(App.scala:80)
  [error]  at scala.App.main$(App.scala:78)
  [error]  at com.sushant.example.PiotrekExample$.main(PiotrekExample.scala:8)
  [error]  at com.sushant.example.PiotrekExample.main(PiotrekExample.scala)
  [error]  Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.7m.dylib, 9): image not found
  [error]   at com.sun.jna.Native.open(Native Method)
  [error]   at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
  [error]   ... 21 more

The same code works with scalapy 0.3.0. Any idea what could be missing from my setup?
It fails on val tf = py.module("tensorflow")

Shadaj Laddad
@shadaj
@Avasil the new ScalaPy release doesn't need Jep anymore, it uses your existing Python installation directly. You can add your installation to the path with javaOptions in Test += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" in your SBT settings
Piotr Gawryś
@Avasil
Thank you, it's working fine now!
Piotr Gawryś
@Avasil

Is there any example of a class which can be created by the user calling constructor directly? e.g.

@py.native class Conv2D(
                       filters: Int,
                       kernel_size: py.|[Int, (Int, Int)],
                       strides: py.|[Int, (Int, Int)] = (1, 1),
                       padding: String = "valid",
                       data_format: String = "channels_last",
                       dilation_rate: py.|[Int, (Int, Int)] = (1, 1),
                       activation: Option[String] = None,
                       use_bias: Boolean = true,
                       kernel_initializer: String = "glorot_uniform",
                       bias_initializer: String = "zeros",
                       kernel_regularizer: Option[String] = None,
                       bias_regularizer: Option[String] = None,
                       activity_regularizer: Option[String] = None,
                       kernel_constraint: Option[String] = None,
                       bias_constraint: Option[String] = None
                       ) extends py.Object

I would like to be able to call it like Conv2D(32, kernel_size = (3, 3)) but I'm not sure how to provide PyValue

Shadaj Laddad
@shadaj
@Avasil there's no class equivalent yet in ScalaPy (this is on my mind but will likely require fairly complex macro transformations), so you can just use objects where the apply method calls a method facade
Piotr Gawryś
@Avasil

@shadaj Thanks, I will try it. Do you mean something like:

@py.native abstract class MyClass(x: String) extends py.Object

@py.native object MyClass extends py.Object {
  def apply(x: String): MyClass = py.native
}

Although I need to provide Reader[MyClass] and PyValue for the object itself
I have some classes working if there is a single parameter (val value: PyValue) but I have trouble to do it if there are more

Shadaj Laddad
@shadaj

@Avasil MyClass would have to be a trait with x being defined as a method inside. Right now, ScalaPy also doesn't support top-level objects as facades, so you would need to define a separate trait which contains the constructor which gets a bit messy. What I'd suggest is defining the constructor inside the facade for the containing module, similar to how https://github.com/shadaj/scalapy-tensorflow/blob/master/src/main/scala/me/shadaj/scalapy/tensorflow/TensorFlow.scala#L52 defines the constructor for Variable.

This is not too elegant unfortunately, but improving this is my top priority after stabilizing the new interpreter backend so should improve soon!

lasimon
@lasimon
Trying version 0.4.0. Updated build.sbt to use the new javaOptions but now I'm getting: linux-x86-64/libpython3.so not found. I have a libpython3.6.so
lasimon
@lasimon
Also a 3.7.so
Shadaj Laddad
@shadaj
@lasimon hmm, that's odd; what's the exact error you're seeing? ScalaPy should try to fall back to other library names when initializing the interpreter
lasimon
@lasimon
@shadaj build.sbt:
fork := true
import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
// javaOptions += "-Djna.library.path=/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib"
[info] running (fork) wumpusWorld.DQNWorld
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] libpython3.so: cannot open shared object file: No such file or directory
[error] Native library (linux-x86-64/libpython3.so) not found in resource path ([file:/tmp/sbt_82d1e13c/job-1/target/55c69968/832b95f1/wumpusworld_2.12-0.1.jar, file:/tmp/sbt_82d1e13c/target/dc7f77ac/b41586e0/scala-library-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/6a07f7c0/c6806aaf/refined_2.12-0.9.15.jar, file:/tmp/sbt_82d1e13c/target/f631ecf1/e854b270/figaro_2.12-5.0.0.0.jar, file:/tmp/sbt_82d1e13c/target/da834eb9/baaae62c/scalapy-core_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/95d6cb8d/f93cbdeb/scala-compiler-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/17f40488/b6b93b90/scala-reflect-2.12.12.jar, file:/tmp/sbt_82d1e13c/target/df126de0/b50eae97/shapeless_2.12-2.3.3.jar, file:/tmp/sbt_82d1e13c/target/906e7fb1/01c9646c/scala-xml_2.12-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/b81ef6b7/795641c2/asm-3.3.1.jar, file:/tmp/sbt_82d1e13c/target/a5cd33ac/662f0474/commons-math3-3.3.jar, file:/tmp/sbt_82d1e13c/target/0421973b/0603d5cb/jsci-1.2.jar, file:/tmp/sbt_82d1e13c/target/d88c303f/81fcec1e/akka-actor_2.12-2.4.18.jar, file:/tmp/sbt_82d1e13c/target/ea9de1d2/c7269f1b/breeze_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/29f545e8/0f3e3243/argonaut_2.12-6.2.jar, file:/tmp/sbt_82d1e13c/target/846ef561/db21d5e7/prefuse-beta-20071021.jar, file:/tmp/sbt_82d1e13c/target/7bc43030/382b5f00/scala-swing_2.12-2.0.0.jar, file:/tmp/sbt_82d1e13c/target/8af72ca7/8c48d6d7/scalapy-macros_2.12-0.4.0.jar, file:/tmp/sbt_82d1e13c/target/2b3b90b1/f37928f3/jna-5.5.0.jar, file:/tmp/sbt_82d1e13c/target/f7ef733c/1d4f9fbf/macro-compat_2.12-1.1.1.jar, file:/tmp/sbt_82d1e13c/target/496371c0/d0443e0c/lpsolve-2003-11-03.jar, file:/tmp/sbt_82d1e13c/target/7546bb71/2faab0fb/mathml-dom-java-2.0.jar, file:/tmp/sbt_82d1e13c/target/4399f739/4f47a0d5/xercesImpl-2.8.1.jar, file:/tmp/sbt_82d1e13c/target/c5d711c7/d61d4b86/config-1.3.0.jar, file:/tmp/sbt_82d1e13c/target/f679673a/b69204cf/scala-java8-compat_2.12-0.8.0.jar, file:/tmp/sbt_82d1e13c/target/84087de9/b5315cdf/breeze-macros_2.12-0.13.1.jar, file:/tmp/sbt_82d1e13c/target/419dc0c7/453aa781/core-1.1.2.jar, file:/tmp/sbt_82d1e13c/target/2ac29a43/764709f2/arpack_combined_all-0.1.jar, file:/tmp/sbt_82d1e13c/target/47c97988/b010f01b/opencsv-2.3.jar, file:/tmp/sbt_82d1e13c/target/cfb0976e/6518a453/jtransforms-2.4.0.jar, file:/tmp/sbt_82d1e13c/target/e10d518a/14229e37/spire_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/cb619cd6/dd21cb48/slf4j-api-1.7.5.jar, file:/tmp/sbt_82d1e13c/target/4a189398/a8781ea4/xml-apis-1.3.03.jar, file:/tmp/sbt_82d1e13c/target/f8a5c36f/ff1b5461/junit-4.8.2.jar, file:/tmp/sbt_82d1e13c/target/84ed269c/8778977b/spire-macros_2.12-0.13.0.jar, file:/tmp/sbt_82d1e13c/target/b67dc7de/2334303a/machinist_2.12-0.6.1.jar])
[error] at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error] at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error] at com.sun.jna.Native.register(Native.java:1722)
[error] at com.sun.jna.Native.register(Native.java:1443)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] at scala.util.Try$.apply(Try.scala:213)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error] at scala.collection.LinearSeqOptimized.find(LinearSeqOptimized.scala:115)
[error] at scala.collection.LinearSeqOptimized.find$(LinearSeqOptimized.scala:112)
[error] at scala.collection.immutable.List.find(List.scala:91)
[error] at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error] at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95

python3-config --ldflags
-L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
lasimon
@lasimon
larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ cd /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
larrysimon@inspiron:/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu$ ls
config.c install-sh libpython3.6m-pic.a libpython3.6.so makesetup python.o Setup.config
config.c.in libpython3.6m.a libpython3.6m.so Makefile python-config.py Setup Setup.local
Shadaj Laddad
@shadaj
Ah, looks like you have Python 3.6, which ScalaPy doesn't load by default (though perhaps it should). For now, could you try adding the environment variable SCALAPY_PYTHON_LIBRARY=python3.6 before launching SBT?
lasimon
@lasimon
larrysimon@inspiron:~/IdeaProjects/wumpus$ echo $SCALAPY_PYTHON_LIBRARY
python3.6
then sbt run
... same error
BTW what I'm doing is a reference implementation of a solution to the classic Russell & Norvig Wumpus World using Deep RL on pyTorch. A Python version is an assignment I set for my Intelligent Agents class at U Toronto Continuing Education.
lasimon
@lasimon
I also have Python 3.7 installed if that's a help (or a conflict).
lasimon
@lasimon
Tried it on my mac with and without setting the env variable and same result (libpython3.dylib image not found). Mac is also Python 3.6.
lasimon
@lasimon
Same thing on another mac running python 3.8
Shadaj Laddad
@shadaj
@lasimon oh, there's actually a big bug in the docs! You should be setting javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" (--prefix, instead of --ldflags). I'll update the docs to fix this shortly
lasimon
@lasimon
@shadaj Hmm. Still not working. config --prefix returns /usr. The .so is in /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/. The directory has two files in it: libpython3.7m.so libpython3.7.so
Shadaj Laddad
@shadaj
hmm, it seems like the python3-config is pointing to a system installation in /usr/lib instead of your python3.7 installation. Do you have a python3.7-config command available?
lasimon
@lasimon
I was able to install it using "sudo apt install python3.7-dev". If I now do "python3.7-config --prefix" it still replies with /usr.
Brandon Stilson
@bbstilson
Hey folks. I'm having trouble running the hello world, likely because my python environment is basically a superfund site.
scalapy-test:>python3-config --ldflags
-L/Users/brandons/.pyenv/versions/3.8.5/lib/python3.8/config-3.8-darwin -lintl -ldl -framework CoreFoundation
When I run the getting started example, i get the following stack trace:
sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>
this is my build.sbt
ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
Brandon Stilson
@bbstilson
actually I made a docker container and ran into the same problem
Shadaj Laddad
@shadaj
@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.
Brandon Stilson
@bbstilson
I got the same error sadly. I'll open an issue.
Travis
@travissarles
I'm getting the same issue as @bbstilson
Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0
Jakub Chrobasik
@jachro
We solved that issue by specifying:
  • VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
  • environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
    I hope that helps someone :)
1 reply
Peter Jansen
@peterjansen_ai_twitter
I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125
Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)
Peter Jansen
@peterjansen_ai_twitter
Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?
Peter Jansen
@peterjansen_ai_twitter
image.png
I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):
Even with the same array content
Peter Jansen
@peterjansen_ai_twitter
Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?
Pedro Luis
@PedroCorreiaLuis
Screenshot 2020-12-18 at 12.58.17.png
I am struggling a lot, can someone help me what should I put in the sbt, thanks!
Shadaj Laddad
@shadaj
@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?
@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now