Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 23 21:47

    shadaj on py-local-docs

    (compare)

  • Oct 23 21:47

    shadaj on main

    Add docs explaining how to use … (compare)

  • Oct 23 21:47
    shadaj closed #231
  • Oct 23 21:47
    shadaj closed #189
  • Oct 23 20:41
    shadaj opened #231
  • Oct 23 20:41
    shadaj milestoned #231
  • Oct 23 20:41

    shadaj on py-local-docs

    Add docs explaining how to use … (compare)

  • Oct 23 19:56
    shadaj milestoned #141
  • Oct 23 19:56
    shadaj demilestoned #141
  • Oct 23 19:56
    shadaj demilestoned #186
  • Oct 23 19:56
    shadaj milestoned #186
  • Oct 23 19:55
    shadaj closed #186
  • Oct 23 19:44

    shadaj on main

    Update sbt-mdoc to 2.2.24 (#229) (compare)

  • Oct 23 19:44
    shadaj closed #229
  • Oct 21 17:03
    scala-steward opened #230
  • Oct 20 15:08
    scala-steward opened #229
  • Oct 18 22:38
    scala-steward synchronize #227
  • Oct 18 22:11

    shadaj on main

    Update sbt-assembly to 1.1.0 (#… (compare)

  • Oct 18 22:11
    shadaj closed #220
  • Oct 18 22:11
    shadaj auto_merge_enabled #220
lasimon
@lasimon
larrysimon@inspiron:~/IdeaProjects/wumpus$ echo $SCALAPY_PYTHON_LIBRARY
python3.6
then sbt run
... same error
BTW what I'm doing is a reference implementation of a solution to the classic Russell & Norvig Wumpus World using Deep RL on pyTorch. A Python version is an assignment I set for my Intelligent Agents class at U Toronto Continuing Education.
lasimon
@lasimon
I also have Python 3.7 installed if that's a help (or a conflict).
lasimon
@lasimon
Tried it on my mac with and without setting the env variable and same result (libpython3.dylib image not found). Mac is also Python 3.6.
lasimon
@lasimon
Same thing on another mac running python 3.8
Shadaj Laddad
@shadaj
@lasimon oh, there's actually a big bug in the docs! You should be setting javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" (--prefix, instead of --ldflags). I'll update the docs to fix this shortly
lasimon
@lasimon
@shadaj Hmm. Still not working. config --prefix returns /usr. The .so is in /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/. The directory has two files in it: libpython3.7m.so libpython3.7.so
Shadaj Laddad
@shadaj
hmm, it seems like the python3-config is pointing to a system installation in /usr/lib instead of your python3.7 installation. Do you have a python3.7-config command available?
lasimon
@lasimon
I was able to install it using "sudo apt install python3.7-dev". If I now do "python3.7-config --prefix" it still replies with /usr.
Brandon Stilson
@bbstilson
Hey folks. I'm having trouble running the hello world, likely because my python environment is basically a superfund site.
scalapy-test:>python3-config --ldflags
-L/Users/brandons/.pyenv/versions/3.8.5/lib/python3.8/config-3.8-darwin -lintl -ldl -framework CoreFoundation
When I run the getting started example, i get the following stack trace:
sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>
this is my build.sbt
ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
Brandon Stilson
@bbstilson
actually I made a docker container and ran into the same problem
Shadaj Laddad
@shadaj
@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.
Brandon Stilson
@bbstilson
I got the same error sadly. I'll open an issue.
Travis
@travissarles
I'm getting the same issue as @bbstilson
Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0
Jakub Chrobasik
@jachro
We solved that issue by specifying:
  • VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
  • environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
    I hope that helps someone :)
1 reply
Peter Jansen
@peterjansen_ai_twitter
I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125
Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)
Peter Jansen
@peterjansen_ai_twitter
Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?
Peter Jansen
@peterjansen_ai_twitter
image.png
I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):
Even with the same array content
Peter Jansen
@peterjansen_ai_twitter
Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?
Pedro Luis
@PedroCorreiaLuis
Screenshot 2020-12-18 at 12.58.17.png
I am struggling a lot, can someone help me what should I put in the sbt, thanks!
Shadaj Laddad
@shadaj
@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?
@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now
Kamil Kloch
@kamilkloch
Thanks for fixing python library fallback issues. Any chance for a bugfix release? :)
Shadaj Laddad
@shadaj
@kamilkloch v0.4.1 should be out very soon!
Kamil Kloch
@kamilkloch
@shadaj Thanks! I take this opportunity to thank you for all your work within the Scala community. Slinky, ScalaPy, who knows what brings 2021 ;) U da man.
Shadaj Laddad
@shadaj
@kamilkloch thanks so much, that means a lot! it's been very exciting to see the projects being helpful to the community!
Kamil Kloch
@kamilkloch

Hello all, the folowing snippet fails to compile:

import me.shadaj.scalapy.numpy.NumPy
import me.shadaj.scalapy.py

val np = py.module("numpy").as[NumPy]

could not find implicit value for evidence parameter of type me.shadaj.scalapy.readwrite.Reader[me.shadaj.scalapy.numpy.NumPy]
  val np = py.module("numpy").as[NumPy]

Any hints?

Also, py.module("numpy") compiles despite missing implicit jep: Jep, whereas implicitly[Jep] fails to compile.
Kamil Kloch
@kamilkloch
Update: it looks that the impiict jep instance is required by the (legacy?) artifact https://mvnrepository.com/artifact/me.shadaj/scalapy_2.12/0.2.0 sucked in with the scalapy-numpy facade.
Shadaj Laddad
@shadaj
@kamilkloch ah, yeah, I need to re-release those bindings for the new ScalaPy release
Anatolii Kmetiuk
@anatoliykmetyuk

Hi! I was following the Getting Started https://scalapy.dev/docs/ instructions, but when I tried running the code example, I got:

[error] Exception in thread "main" java.lang.ExceptionInInitializerError
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]     at Main$package$.hello(Main.scala:5)
[error]     at hello.main(Main.scala:4)
[error] Caused by: java.lang.Exception: Unable to locate Python library, tried python3, python3.7, python3.7m
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:19)
[error]     at scala.Option.getOrElse(Option.scala:201)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:19)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:102)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:102)
[error]     ... 4 more

Python was installed via homebrew, the OS is MacOS Catalina. The output of python3-config --configdir is /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin. When I run ls -1 $(python3-config --configdir), I get:

Makefile
Setup
Setup.local
config.c
config.c.in
install-sh
libpython3.9.a
libpython3.9.dylib
makesetup
python-config.py
python.o

So probably javaOptions += s"-Djava.library.path=${"python3-config --configdir".!!.trim}/lib" in the SBT settings cannot locate lib. How do I fix this?

Shadaj Laddad
@shadaj
@anatoliykmetyuk try adding SCALAPY_PYTHON_LIBRARY=python3.9 to your build to override the python version. I'll be publishing an update to the docs with guidance on this situation in just a bit!
Anatolii Kmetiuk
@anatoliykmetyuk
I tried doing so, this did not solve the problem but the error message got changed to include that version of python:
Caused by: java.lang.Exception: Unable to locate Python library, tried python3.9, python3, python3.7, python3.7
implisci
@implisci
Pre-Newbie here (as in not even used ScalaPy). Apologies if my questions are incorrect or assume wrong things. 1) Does ScalaPy load the PY interpreter(s) in an external process(es) and pass python code to them and await results? 2) The difference between something like Tensorflow for Scala and ScalaPy is that the former targets one library but this is (in principle) general? Thanks. Tensorflow if I recall might be in C/C++ with Python API so the second question may not be correct. Thanks.
Shadaj Laddad
@shadaj
@anatoliykmetyuk try python3.9m? at least on my macOS system that one works
@implisci yeah, that's generally the idea. ScalaPy actually integrates with the Python C APIs, though, so it can avoid generating Python code and have much faster interop. And 2 is exactly right, the goal is to be able to use existing Python libraries that often have a lot of logic baked in which is hard to port to Scala
Julian Pieles
@razem-io
Hi, perhaps a stupid question but I can't find how to represent the following:
from abc import xyz
Julian Pieles
@razem-io
What I tried so far: val pylon = py.module("abc", "xyz")
Julian Pieles
@razem-io
What works is: val pylon = py.module("abc") but this is just an empty init.py