Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 16 16:13
    kiendang commented #309
  • Sep 16 13:40
    alexarchambault commented #309
  • Sep 16 13:40
    alexarchambault commented #309
  • Sep 15 15:48
    kiendang commented #309
  • Sep 15 15:47
    kiendang commented #309
  • Sep 15 15:22
    alexarchambault commented #309
  • Sep 15 15:19
    alexarchambault commented #309
  • Sep 15 14:01
    alexarchambault commented #309
  • Sep 15 14:01
    alexarchambault opened #309
  • Sep 07 20:42
    scala-steward opened #308
  • Sep 06 16:13
    kiendang synchronize #303
  • Sep 06 09:31
    kiendang commented #303
  • Sep 06 09:30
    kiendang opened #307
  • Sep 06 09:15
    kiendang commented #303
  • Sep 06 09:00
    kiendang synchronize #303
  • Sep 01 19:45
    scala-steward opened #306
  • Sep 01 15:46
    kiendang synchronize #303
  • Sep 01 05:46
    kiendang synchronize #303
  • Sep 01 05:43
    kiendang synchronize #303
  • Sep 01 04:51
    kiendang synchronize #303
Brandon Stilson
@bbstilson
sbt:scalapy-test> run
[info] running (fork) Main 
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
[error] dlopen(libpython3.dylib, 9): image not found
[error] dlopen(libpython3.dylib, 9): image not found
[error] Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]         at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error]         at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error]         at com.sun.jna.Native.register(Native.java:1722)
[error]         at com.sun.jna.Native.register(Native.java:1443)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:13)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error]         at scala.util.Try$.apply(Try.scala:210)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$1$adapted(CPythonAPI.scala:13)
[error]         at scala.collection.immutable.List.find(List.scala:413)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:13)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:95)
[error]         at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]         at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]         at Main$.main(Main.scala:13)
[error]         at Main.main(Main.scala)
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
[error]                 ... 16 more
[error]         Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
[error]                 at com.sun.jna.Native.open(Native Method)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
[error]                 ... 16 more
[error]         Suppressed: java.io.IOException: Native library (darwin/libpython3.dylib) not found in resource path (TOO LONG)
[error]                 at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
[error]                 at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
[error]                 ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Nov 24, 2020 2:04:42 PM
sbt:scalapy-test>
this is my build.sbt
ThisBuild / scalaVersion := "2.13.3"

libraryDependencies += "me.shadaj" %% "scalapy-core" % "0.4.0"

fork := true

import scala.sys.process._
javaOptions += s"-Djava.library.path=${"python3-config --ldflags".!! + "/lib"}"
Brandon Stilson
@bbstilson
actually I made a docker container and ran into the same problem
Shadaj Laddad
@shadaj
@bbstilson could you try modifying your SBT build to use python3-config --prefix instead of python3-config --ldflags? There was a bug in the JVM setup instructions on the website.
Brandon Stilson
@bbstilson
I got the same error sadly. I'll open an issue.
Travis
@travissarles
I'm getting the same issue as @bbstilson
Mac OS 10.15, Scala 2.13.4, scalapy-core 0.4.0
Jakub Chrobasik
@jachro
We solved that issue by specifying:
  • VM Option -Djna.library.path=/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib which is effectively -Djna.library.path=${"python3-config --prefix".!!.trim}/lib
  • environment variable: SCALAPY_PYTHON_LIBRARY = python3.8
    I hope that helps someone :)
1 reply
Peter Jansen
@peterjansen_ai_twitter
I have a similar issue -- complicated slightly by using a conda environment. I posted my (current) solution to the github issue: shadaj/scalapy#125
Though it runs on the command line, it's still throwing the issue when running from intellij, which I don't fully appreciate why (they're using the same build.sbt, etc -- perhaps it's something with an environment variable)
Peter Jansen
@peterjansen_ai_twitter
Is there a fast, cannonical way of packing a multidimensional scala array (e.g. of Ints) into a numpy array?
Peter Jansen
@peterjansen_ai_twitter
image.png
I've put together a fairly hacky way of doing things based off reading the tests, but it's not fast (the array is only ~300x300x3 and takes 500msec to convert) -- but, interestingly, the time appears to be non deterministic (it ranges between 400-4000msec):
Even with the same array content
Peter Jansen
@peterjansen_ai_twitter
Interesting... it also segfaults when using matplotlib to plot that small 300x300x3 image, but not right away. After ~20-30 attempts in a loop, it slows down, then eventually segfaults. I seem to be able to force a segfault by calling System.gc() right before the call to matplotlib to plot the image. I wonder if the converted n-d array is being prematurely garbage collected somehow?
Pedro Luis
@PedroCorreiaLuis
Screenshot 2020-12-18 at 12.58.17.png
I am struggling a lot, can someone help me what should I put in the sbt, thanks!
Shadaj Laddad
@shadaj
@peterjansen_ai_twitter unfortunately converting multidimensional arrays requires a bunch of maps right now, though I hope to improve that by making toPythonCopy/toPythonProxy aware of such types; the segfault is surprising, are you running on the JVM?
@PedroCorreiaLuis this is a bug that will be fixed in the next release of ScalaPy, you can add the environment variable SCALAPY_PYTHON_LIBRARY=python3.6m (replacing 3.6 with the appropriate Python version) to fix it for now
Kamil Kloch
@kamilkloch
Thanks for fixing python library fallback issues. Any chance for a bugfix release? :)
Shadaj Laddad
@shadaj
@kamilkloch v0.4.1 should be out very soon!
Kamil Kloch
@kamilkloch
@shadaj Thanks! I take this opportunity to thank you for all your work within the Scala community. Slinky, ScalaPy, who knows what brings 2021 ;) U da man.
Shadaj Laddad
@shadaj
@kamilkloch thanks so much, that means a lot! it's been very exciting to see the projects being helpful to the community!
Kamil Kloch
@kamilkloch

Hello all, the folowing snippet fails to compile:

import me.shadaj.scalapy.numpy.NumPy
import me.shadaj.scalapy.py

val np = py.module("numpy").as[NumPy]

could not find implicit value for evidence parameter of type me.shadaj.scalapy.readwrite.Reader[me.shadaj.scalapy.numpy.NumPy]
  val np = py.module("numpy").as[NumPy]

Any hints?

Also, py.module("numpy") compiles despite missing implicit jep: Jep, whereas implicitly[Jep] fails to compile.
Kamil Kloch
@kamilkloch
Update: it looks that the impiict jep instance is required by the (legacy?) artifact https://mvnrepository.com/artifact/me.shadaj/scalapy_2.12/0.2.0 sucked in with the scalapy-numpy facade.
Shadaj Laddad
@shadaj
@kamilkloch ah, yeah, I need to re-release those bindings for the new ScalaPy release
Anatolii Kmetiuk
@anatoliykmetyuk

Hi! I was following the Getting Started https://scalapy.dev/docs/ instructions, but when I tried running the code example, I got:

[error] Exception in thread "main" java.lang.ExceptionInInitializerError
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
[error]     at Main$package$.hello(Main.scala:5)
[error]     at hello.main(Main.scala:4)
[error] Caused by: java.lang.Exception: Unable to locate Python library, tried python3, python3.7, python3.7m
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$new$2(CPythonAPI.scala:19)
[error]     at scala.Option.getOrElse(Option.scala:201)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:19)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:102)
[error]     at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:102)
[error]     ... 4 more

Python was installed via homebrew, the OS is MacOS Catalina. The output of python3-config --configdir is /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin. When I run ls -1 $(python3-config --configdir), I get:

Makefile
Setup
Setup.local
config.c
config.c.in
install-sh
libpython3.9.a
libpython3.9.dylib
makesetup
python-config.py
python.o

So probably javaOptions += s"-Djava.library.path=${"python3-config --configdir".!!.trim}/lib" in the SBT settings cannot locate lib. How do I fix this?

Shadaj Laddad
@shadaj
@anatoliykmetyuk try adding SCALAPY_PYTHON_LIBRARY=python3.9 to your build to override the python version. I'll be publishing an update to the docs with guidance on this situation in just a bit!
Anatolii Kmetiuk
@anatoliykmetyuk
I tried doing so, this did not solve the problem but the error message got changed to include that version of python:
Caused by: java.lang.Exception: Unable to locate Python library, tried python3.9, python3, python3.7, python3.7
implisci
@implisci
Pre-Newbie here (as in not even used ScalaPy). Apologies if my questions are incorrect or assume wrong things. 1) Does ScalaPy load the PY interpreter(s) in an external process(es) and pass python code to them and await results? 2) The difference between something like Tensorflow for Scala and ScalaPy is that the former targets one library but this is (in principle) general? Thanks. Tensorflow if I recall might be in C/C++ with Python API so the second question may not be correct. Thanks.
Shadaj Laddad
@shadaj
@anatoliykmetyuk try python3.9m? at least on my macOS system that one works
@implisci yeah, that's generally the idea. ScalaPy actually integrates with the Python C APIs, though, so it can avoid generating Python code and have much faster interop. And 2 is exactly right, the goal is to be able to use existing Python libraries that often have a lot of logic baked in which is hard to port to Scala
Julian Pieles
@razem-io
Hi, perhaps a stupid question but I can't find how to represent the following:
from abc import xyz
Julian Pieles
@razem-io
What I tried so far: val pylon = py.module("abc", "xyz")
Julian Pieles
@razem-io
What works is: val pylon = py.module("abc") but this is just an empty init.py
Julian Pieles
@razem-io
nvm -> py.module("abc.xyz")
RndMnkIII
@RndMnkIII

Hello, I am trying to see how ScalaPy works through a simple example to visualize a Jupyter widget using the python module of ipywidgets, with the following code:

import $ivy.`me.shadaj::scalapy-core:0.4.2`

import me.shadaj.scalapy.py
import me.shadaj.scalapy.py.SeqConverters

val w = py.module ("ipywidgets")
w: py.Module = <module 'ipywidgets' from '/home/javier/anaconda3/envs/jupyterlab-env/lib/python3.8/site-packages/ipywidgets/__init__.py'>
val dsp = py.module("IPython.display")
dsp: py.Module = <module 'IPython.display' from '/home/javier/anaconda3/envs/jupyterlab-env/lib/python3.8/site-packages/IPython/display.py'>
val w1 = w.IntSlider(10)
w1: py.Dynamic = IntSlider(value=10)

dsp.display(w1.toPythonProxy)

But I don't know how to pass the reference of the python IntSlider object to the python display function. I have tried with the toPythonProxy function but it is not defined for the IntSlider widget object and it produces an error:

me.shadaj.scalapy.py.PythonException: <class 'AttributeError'> 'IntSlider' object has no attribute 'toPythonProxy'
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:315)
  me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:301)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$select$1(CPythonInterpreter.scala:501)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:148)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.select(CPythonInterpreter.scala:493)
  me.shadaj.scalapy.py.AnyDynamics.selectDynamic(Dynamic.scala:43)
  me.shadaj.scalapy.py.AnyDynamics.selectDynamic$(Dynamic.scala:42)
  me.shadaj.scalapy.py.AnyDynamics$$anon$7$$anon$8.selectDynamic(Dynamic.scala:31)
  ammonite.$sess.cmd9$Helper.<init>(cmd9.sc:1)
  ammonite.$sess.cmd9$.<clinit>(cmd9.sc:7)

How could I pass the IntSlider (py.Dynamic) object to the python display function?

if I write it as dsp.display(w1) prints out: res11: py.Dynamic = None
shadaj
@shadaj:matrix.org
[m]
@RndMnkIII: aha, so right now Python widgets don't show up in Jupyter, since Almond takes over as the interface to the frontend and so the Python-Jupyter integration is not passed through. However, fixing this is actually a planned project for this summer's Google Summer of Code, which ScalaPy is participating in as part of the Scala Center, so there will hopefully be more progress on this soon!
But generally, toPythonProxy and toPythonCopy only need to be used when passing Scala sequences into Python code (an explicit conversion is required because these operations can be fairly costly on the JVM).
Cedric John
@cedricmjohn
Hi there @shadaj:matrix.org . First of all, amazingly good idea: I definitely want to be able to use Python from the comfort of Scala! This would be the best of both worlds. But after trying for 2 days to make this work on my MacBook Pro, I am somewhat desperate. I have very similar issues to what @lasimon had. I tried multiple combination of commands (I believe the docs is still not up to date?) and I never manage to get my Python module to run. I have tried it on the Almond kernel on Jupyter, but also on IntelliJIdea and a simple SBT file. No matter what I tried I don't get it. Can you help please?
Issues that I am having fall into two categories:
1] Either I get a java.lang.UnsatisfiedLinkError: Unable to load library 'python3' error as the system cannot find the 'darwin/libpython3.dylib'.
2] Or in rare occasions I ran into an incompatible JNA version.
I think whatever I give as my javaOptions string is not correct. I tried the solutions you offered @shadaj:matrix.org but none worked. I found an actual 'libpython3.7.dylib' and give the path to this lib folder as my argument, but no idea why the Scalapy library is trying to load a generic 'libpython3.dylib'?
Clearly, I don't understand what I am doing but I also cannot find any clear documentation for this issue. Appreciate any help anyone can get me. Thanks!
shadaj
@shadaj:matrix.org
[m]
Hi @cedricmjohn! The latest docs at https://scalapy.dev/docs/ should work. It's interesting that you're seeing python3 fail to load, since ScalaPy has logic (in 0.4.2) to fall back to other Python versions (https://github.com/shadaj/scalapy/blob/master/core/jvm/src/main/scala/me/shadaj/scalapy/interpreter/CPythonAPI.scala#L10). Can you ensure that you're on the latest version? This behavior used to be buggy in previous versions.
Cedric John
@cedricmjohn
Hi @shadaj:matrix.org ! Unfortunately I am following the exact steps in your latest version of the tutorial, and I copied the file exactly as shown in my sbt file. I have Python 3.8.5 installed via Anaconda python on MacOS BigSur. I attached below the exact runtime error I am getting:

/Library/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=58295:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/cedric/IdeaProjects/scalapy-test/target/scala-2.13/classes:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-core_2.13/0.4.2/scalapy-core_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-macros_2.13/0.4.2/scalapy-macros_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.5/scala-library-2.13.5.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.5/scala-reflect-2.13.5.jar main
java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
dlopen(libpython3.dylib, 9): image not found
dlopen(libpython3.dylib, 9): image not found
dlopen(/Users/cedric/Library/Frameworks/python3.framework/python3, 9): image not found
dlopen(/Library/Frameworks/python3.framework/python3, 9): image not found
dlopen(/System/Library/Frameworks/python3.framework/python3, 9): image not found
Native library (darwin/libpython3.dylib) not found in resource path (/Users/cedric/IdeaProjects/scalapy-test/target/scala-2.13/classes:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-core_2.13/0.4.2/scalapy-core_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-macros_2.13/0.4.2/scalapy-macros_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.5/scala-library-2.13.5.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.5/scala-reflect-2.13.5.jar)
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:301)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:461)
at com.sun.jna.Native.register(Native.java:1722)
at com.sun.jna.Native.register(Native.java:1443)
at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$loadAttempts$1(CPythonAPI.scala:20)
at scala.collection.immutable.Stream.map(Stream.scala:173)
at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:19)
at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:111)
at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:111)
at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
at main$.delayedEndpoint$main$1(main.scala:5)
at main$delayedInit$body.apply(main.scala:1)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:76)
at scala.App.$anonfun$main$1$adapted(App.scala:76)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at scala.App.main(App.scala:76)
at scala.App.main$(App.scala:74)
at main$.main(main.scala:1)
at main.main(main.scala)
Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
at com.sun.jna.Native.open(Native Method)
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
... 24 more
Suppr
Any help really welcomed as I would LOVE to run this library. It's probably a configuration problem between my Python and Java install, but I cannot figure it out.
Cedric John
@cedricmjohn
By the way, when I run the code you give in the SBT file as an independent piece of Scala code I get the following directory name for my Python library: "/Applications/anaconda3/lib/python3.8/config-3.8-darwin". I can confirm that there are no 'Python3.dlib' file at this location. So it seems that it is the SBT code that is not pointing to the relevant folder in my system?
Mykola Medynskyi
@jlareckm_twitter

Hi @cedricmjohn, 
It seems to me that I had a similar problem to yours (during running the scalaPy in intellij), and in my case, it was solved in the next way: 
1. firstly run python3-config --ldflags
You maybe will see something like this: -L/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

  1. Then run command ls /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin

and check if the libpython3.9.dylib exists

  1. If it exists specify the jna.library.path in VM parameters of configurations in intellij. In my case it looked like this: -Djna.library.path=/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin

Maybe it will be helpful

Screenshot 2021-04-01 at 23.27.54.png