Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 09:23
    jlareck synchronize #194
  • Jun 12 21:54
    jlareck synchronize #194
  • Jun 11 18:29
    jlareck ready_for_review #194
  • Jun 11 12:24
    jlareck commented #194
  • Jun 11 12:08
    jlareck synchronize #194
  • Jun 09 12:32
    Ang9876 edited #195
  • Jun 09 12:32
    Ang9876 edited #195
  • Jun 09 12:28
    Ang9876 opened #195
  • Jun 08 22:40
    shadaj converted_to_draft #194
  • Jun 08 22:19
    jlareck synchronize #194
  • Jun 08 22:14
    jlareck opened #194
  • Jun 07 08:14
    scala-steward opened #193
  • Jun 02 16:44
    shadaj opened #192
  • Jun 02 16:28
    shadaj commented #188
  • Jun 02 16:28
    shadaj commented #188
  • Jun 02 03:22
    shadaj opened #191
  • Jun 02 03:22
    shadaj labeled #191
  • Jun 02 03:20
    shadaj edited #190
  • Jun 02 03:19
    shadaj labeled #190
  • Jun 02 03:19
    shadaj opened #190
RndMnkIII
@RndMnkIII

Hello, I am trying to see how ScalaPy works through a simple example to visualize a Jupyter widget using the python module of ipywidgets, with the following code:

import $ivy.`me.shadaj::scalapy-core:0.4.2`

import me.shadaj.scalapy.py
import me.shadaj.scalapy.py.SeqConverters

val w = py.module ("ipywidgets")
w: py.Module = <module 'ipywidgets' from '/home/javier/anaconda3/envs/jupyterlab-env/lib/python3.8/site-packages/ipywidgets/__init__.py'>
val dsp = py.module("IPython.display")
dsp: py.Module = <module 'IPython.display' from '/home/javier/anaconda3/envs/jupyterlab-env/lib/python3.8/site-packages/IPython/display.py'>
val w1 = w.IntSlider(10)
w1: py.Dynamic = IntSlider(value=10)

dsp.display(w1.toPythonProxy)

But I don't know how to pass the reference of the python IntSlider object to the python display function. I have tried with the toPythonProxy function but it is not defined for the IntSlider widget object and it produces an error:

me.shadaj.scalapy.py.PythonException: <class 'AttributeError'> 'IntSlider' object has no attribute 'toPythonProxy'
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:315)
  me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:301)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$select$1(CPythonInterpreter.scala:501)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:148)
  me.shadaj.scalapy.interpreter.CPythonInterpreter$.select(CPythonInterpreter.scala:493)
  me.shadaj.scalapy.py.AnyDynamics.selectDynamic(Dynamic.scala:43)
  me.shadaj.scalapy.py.AnyDynamics.selectDynamic$(Dynamic.scala:42)
  me.shadaj.scalapy.py.AnyDynamics$$anon$7$$anon$8.selectDynamic(Dynamic.scala:31)
  ammonite.$sess.cmd9$Helper.<init>(cmd9.sc:1)
  ammonite.$sess.cmd9$.<clinit>(cmd9.sc:7)

How could I pass the IntSlider (py.Dynamic) object to the python display function?

if I write it as dsp.display(w1) prints out: res11: py.Dynamic = None
shadaj
@shadaj:matrix.org
[m]
@RndMnkIII: aha, so right now Python widgets don't show up in Jupyter, since Almond takes over as the interface to the frontend and so the Python-Jupyter integration is not passed through. However, fixing this is actually a planned project for this summer's Google Summer of Code, which ScalaPy is participating in as part of the Scala Center, so there will hopefully be more progress on this soon!
But generally, toPythonProxy and toPythonCopy only need to be used when passing Scala sequences into Python code (an explicit conversion is required because these operations can be fairly costly on the JVM).
Cedric John
@cedricmjohn
Hi there @shadaj:matrix.org . First of all, amazingly good idea: I definitely want to be able to use Python from the comfort of Scala! This would be the best of both worlds. But after trying for 2 days to make this work on my MacBook Pro, I am somewhat desperate. I have very similar issues to what @lasimon had. I tried multiple combination of commands (I believe the docs is still not up to date?) and I never manage to get my Python module to run. I have tried it on the Almond kernel on Jupyter, but also on IntelliJIdea and a simple SBT file. No matter what I tried I don't get it. Can you help please?
Issues that I am having fall into two categories:
1] Either I get a java.lang.UnsatisfiedLinkError: Unable to load library 'python3' error as the system cannot find the 'darwin/libpython3.dylib'.
2] Or in rare occasions I ran into an incompatible JNA version.
I think whatever I give as my javaOptions string is not correct. I tried the solutions you offered @shadaj:matrix.org but none worked. I found an actual 'libpython3.7.dylib' and give the path to this lib folder as my argument, but no idea why the Scalapy library is trying to load a generic 'libpython3.dylib'?
Clearly, I don't understand what I am doing but I also cannot find any clear documentation for this issue. Appreciate any help anyone can get me. Thanks!
shadaj
@shadaj:matrix.org
[m]
Hi @cedricmjohn! The latest docs at https://scalapy.dev/docs/ should work. It's interesting that you're seeing python3 fail to load, since ScalaPy has logic (in 0.4.2) to fall back to other Python versions (https://github.com/shadaj/scalapy/blob/master/core/jvm/src/main/scala/me/shadaj/scalapy/interpreter/CPythonAPI.scala#L10). Can you ensure that you're on the latest version? This behavior used to be buggy in previous versions.
Cedric John
@cedricmjohn
Hi @shadaj:matrix.org ! Unfortunately I am following the exact steps in your latest version of the tutorial, and I copied the file exactly as shown in my sbt file. I have Python 3.8.5 installed via Anaconda python on MacOS BigSur. I attached below the exact runtime error I am getting:

/Library/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=58295:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/cedric/IdeaProjects/scalapy-test/target/scala-2.13/classes:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-core_2.13/0.4.2/scalapy-core_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-macros_2.13/0.4.2/scalapy-macros_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.5/scala-library-2.13.5.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.5/scala-reflect-2.13.5.jar main
java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
dlopen(libpython3.dylib, 9): image not found
dlopen(libpython3.dylib, 9): image not found
dlopen(/Users/cedric/Library/Frameworks/python3.framework/python3, 9): image not found
dlopen(/Library/Frameworks/python3.framework/python3, 9): image not found
dlopen(/System/Library/Frameworks/python3.framework/python3, 9): image not found
Native library (darwin/libpython3.dylib) not found in resource path (/Users/cedric/IdeaProjects/scalapy-test/target/scala-2.13/classes:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-core_2.13/0.4.2/scalapy-core_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/me/shadaj/scalapy-macros_2.13/0.4.2/scalapy-macros_2.13-0.4.2.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.6.0/jna-5.6.0.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.5/scala-library-2.13.5.jar:/Users/cedric/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.5/scala-reflect-2.13.5.jar)
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:301)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:461)
at com.sun.jna.Native.register(Native.java:1722)
at com.sun.jna.Native.register(Native.java:1443)
at me.shadaj.scalapy.interpreter.CPythonAPIInterface.$anonfun$loadAttempts$1(CPythonAPI.scala:20)
at scala.collection.immutable.Stream.map(Stream.scala:173)
at me.shadaj.scalapy.interpreter.CPythonAPIInterface.<init>(CPythonAPI.scala:19)
at me.shadaj.scalapy.interpreter.CPythonAPI$.<init>(CPythonAPI.scala:111)
at me.shadaj.scalapy.interpreter.CPythonAPI$.<clinit>(CPythonAPI.scala:111)
at me.shadaj.scalapy.interpreter.CPythonInterpreter$.<clinit>(CPythonInterpreter.scala:9)
at me.shadaj.scalapy.py.package$.<clinit>(package.scala:15)
at main$.delayedEndpoint$main$1(main.scala:5)
at main$delayedInit$body.apply(main.scala:1)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:76)
at scala.App.$anonfun$main$1$adapted(App.scala:76)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at scala.App.main(App.scala:76)
at scala.App.main$(App.scala:74)
at main$.main(main.scala:1)
at main.main(main.scala)
Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libpython3.dylib, 9): image not found
at com.sun.jna.Native.open(Native Method)
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
... 24 more
Suppr
Any help really welcomed as I would LOVE to run this library. It's probably a configuration problem between my Python and Java install, but I cannot figure it out.
Cedric John
@cedricmjohn
By the way, when I run the code you give in the SBT file as an independent piece of Scala code I get the following directory name for my Python library: "/Applications/anaconda3/lib/python3.8/config-3.8-darwin". I can confirm that there are no 'Python3.dlib' file at this location. So it seems that it is the SBT code that is not pointing to the relevant folder in my system?
Mykola Medynskyi
@jlareckm_twitter

Hi @cedricmjohn, 
It seems to me that I had a similar problem to yours (during running the scalaPy in intellij), and in my case, it was solved in the next way: 
1. firstly run python3-config --ldflags
You maybe will see something like this: -L/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

  1. Then run command ls /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin

and check if the libpython3.9.dylib exists

  1. If it exists specify the jna.library.path in VM parameters of configurations in intellij. In my case it looked like this: -Djna.library.path=/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin

Maybe it will be helpful

Screenshot 2021-04-01 at 23.27.54.png
Screenshot 2021-04-01 at 23.28.25.png
I also attached the screenshots how it looked like
Cedric John
@cedricmjohn
Hi @jlareckm_twitter ! Thanks a lot for these detailed explanation, it helps (though the windows are a bit too small for me to see). One issue is that in the /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin I don't have a libpython3.9.dylib file. I completely reinstalled my Python using Anaconda but the problem persists. I found the file in /Application/anaconda3/lib and gave this path, but I have the same error again. Any idea?
Mykola Medynskyi
@jlareckm_twitter
Oh I see @cedricmjohn , in my case I used brew install python3 to install python. Maybe you can try to install python with homebrew but it will possibly have conflicts with anaconda's python (I am not completely sure in i), so maybe it is better to uninstall it before installing python with homebrew
Cedric John
@cedricmjohn
Thanks @jlareckm_twitter . I found the "libpython3.8.dylib" in Anaconda following the post from @shadaj:matrix.org but this did not sort it. Honestly don't know what to do: my scala code keeps trying to find the "libpython3.dylib" when it should be looking for a file named "libpython3.8.dylib". Is this normal @shadaj:matrix.org ? Also how exactly do you add "SCALAPY_PYTHON_LIBRARY=python3.8" to your build? It is not valid SBT syntax so I assume it is not in the sbt file?
Eric K Richardson
@ekrich
You can export SCALAPY_PYTHON_LIBRARY=python3.8 prior to running sbt if that is what they are asking for.
Cedric John
@cedricmjohn

@ekrich @jlareckm_twitter and @shadaj:matrix.org : I figured it out! @jlareckm_twitter your thread was a great help, thanks a lot! I found out thanks to your image where to change the VM options and even where to put the environmental variable, and it worked in my IntelliJ version! Thanks a ton!!!

@shadaj:matrix.org and anyone out there with an Anaconda install: it might be just me, but the 'python3-config --ldflags' approach does not point to the right folder on the latest version of Anaconda Python on a Mac. It does point to the correct Python version in the Darwin folder, but this contains what I think is a symbolic link to the library (libpython3.8.a). It does not work with Scalapy. The trick that worked for me was to redict the path manually as a VM option straight to '/Applications/anaconda3/lib' where the correct 'libpython3.8.dylib' exists.

Is it me and my particular Python install, or does anyone else have this issue too? And I hate to ask it, but how do I set this up on Almond, which was my initial idea?

Thanks and sorry for being a Gitter pain! :)

1 reply
Cedric John
@cedricmjohn

So, I am playing around on Almond with Scala 13.5 and Python 3.8. I encountered an issue with matplotlib. I first do this import:
val plt = py.module("matplotlib.pyplot")
The I execute:
plt.scatter(Seq(1,2,3).toPythonProxy, Seq(1,2,3).toPythonProxy)
But my JVM crashes and I get this error:

"Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!"

Anyone else has this issue? Can we not use libraries with GUI outputs, or maybe we need to do something in addition to that?

Eric K Richardson
@ekrich
@cedricmjohn Great that you put what you found here as well. If you are really motivated you can always put in a PR to improve the docs.
BISHNU SHANKAR
@bishnushankar95_twitter
Is there any way to run a python file using scalapy?
BISHNU SHANKAR
@bishnushankar95_twitter
Hey All,
I was testing parallel execution of scalapy with multiple instances of Module in parallel. My module uses CPython libraries like pandas and NumPy. I was reusing the module instance and using the applyDynamic method in each iteration. I noticed that there are over 10,000 files opened by the process. Is it expected behavior?
Eric K Richardson
@ekrich
@shadaj Have you seen this? https://docs.hpyproject.org/en/latest/index.html Check out handles.
Eddie Hatfield
@e-hat
hey guys -- i'm a student doing CS + math. I love scala + language runtimes + FP. i'm probably not much of a user of this project, but I would love to contribute and learn how real world scala projects are setup. i look forward to working with you all!
Eric K Richardson
@ekrich
Hey @e-hat - not to recruit you away from this project but there is also Scala Native if you like the bare metal stuff with C and garbage collector etc. https://github.com/scala-native/scala-native or Scala.js if you like the front end JavaScript stuff. https://github.com/scala-js/scala-js
shadaj
@shadaj:matrix.org
[m]
@cedricmjohn: hmm, that's interesting to see; I wonder if the JVM's built-in support for GUIs is interfering with Python somehow. In theory GUI apps should work as-is, could you try running the code with Scala Native and see if that works? That'll tell us whether it's a ScalaPy core issue or something specific to the JVM.
@bishnushankar95_twitter: even if you're loading the same module multiple times, Python should be caching the loaded modules, so my guess is that Pandas might be creating some temporary files locally? (though the number of files seems quite large even for that)
@ekrich: yes!! I've been watching HPy closely since it seems to offer a much nicer API for interactive with Python values. So far, though, HPy is more focused on implementing extensions so it's not clear how well it would work in the embedding case.
@e-hat: have you considered applying to Google Summer of Code (https://summerofcode.withgoogle.com/organizations/5449337515016192/)? ScalaPy, Scala Native, and Scala.js are all participating so you might find a project that aligns well with your interests there!
Eric K Richardson
@ekrich
@shadaj:matrix.org that is a great idea for @e-hat - there are lots of nice projects. I think the proposals are due very soon though.
@shadaj:matrix.org Denys pointed the project out here. :point_up: April 8, 2021 9:23 AM and thus we have scala-native/scala-native#2248
Eddie Hatfield
@e-hat
@shadaj:matrix.org @ekrich Thanks for linking those projects! Gotta love the friendly scala community. And I'll check out GSoC and maybe send an application in -- I am working full time this summer so IDK if I can handle GSoC on top of that, but we'll see. Can't hurt to apply.
Eric K Richardson
@ekrich
:thumbsup:
BISHNU SHANKAR
@bishnushankar95_twitter
I have a function def execute(a: String)(params:Any*): py.Dynamic = py.native
How to access the params in the python File. I am not able to understand how to use params. Can anyone help me with this? @shadaj:matrix.org
It will be helpful if the answer has a python function where I can access the params
scalavision
@scalavision
hello everyone, does it exist any scalapy wrapper for numpy, pytorch etc? Maybe there exists something with scala-native?
shadaj
@shadaj:matrix.org
[m]
@scalavision: at the moment, there aren't well-maintained static-type wrappers for those libraries (though this is something that will be improved soon with static type generation!); you can still use these libraries with the dynamically typed API in the meantime
Scala Native is more of a backend for ScalaPy, in terms of compatibility all Python libraries can be used the same way from either the JVM or Scala Native
1 reply
William Narmontas
@ScalaWilliam
Hi, how can you call 'del' in Scalapy?
shadaj
@shadaj:matrix.org
[m]
@ScalaWilliam: shadaj/scalapy#175 :)
William Narmontas
@ScalaWilliam
<3 :-) does it include 'del x' where x is a plain object not a dict? for what I'm doing I need to 'del' to delete a process (not my api!)
I did:
CPythonInterpreter.set("something", Writer.anyWriter[py.Any].write(something.as[py.Any]))
CPythonInterpreter.eval("delete something")
shadaj
@shadaj:matrix.org
[m]
ah, it doesn't include that; I'll have to think about how to implement that, since there isn't an equivalent Python API to that function, but the del method should be triggered by cleanup on a reference count decrement
it might be possible to offer an unsafe API on py.Any that lets you release a Python value even if it's still held by Scala code
William Narmontas
@ScalaWilliam
coool! :-)
shadaj
@shadaj:matrix.org
[m]
that would let you directly drop the reference instead of being at the mercy of the garbage collector
shadaj
@shadaj:matrix.org
[m]
@ScalaWilliam: just put up shadaj/scalapy#177 to resolve the missing APIs for your use-case!
William Narmontas
@ScalaWilliam
amazing @shadaj:matrix.org :-)
BISHNU SHANKAR
@bishnushankar95_twitter

I am trying to run scalapy from intellij in my window machine. I am using anaconda python with python version 3.8.5 . I am getting the below error
`java.lang.UnsatisfiedLinkError: Unable to load library 'python3.8m':
The specified module could not be found.

The specified module could not be found.

The specified module could not be found.

Native library (win32-x86-64/python3.8m.dll) not found in resource path `

shadaj
@shadaj:matrix.org
[m]
@bishnushankar95_twitter: did you make sure to activate the conda environment before launching SBT? (there can be some trickery required when using IDEs, which may launch SBT in the wrong environment); generally ScalaPy should work fine with conda, can you run python-config --ldflags to check where the libraries are installed?