shadaj/scalapy

Cedric John

@cedricmjohn

So, I am playing around on Almond with Scala 13.5 and Python 3.8. I encountered an issue with matplotlib. I first do this import:
val plt = py.module("matplotlib.pyplot")
The I execute:
plt.scatter(Seq(1,2,3).toPythonProxy, Seq(1,2,3).toPythonProxy)
But my JVM crashes and I get this error:

"Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!"

Anyone else has this issue? Can we not use libraries with GUI outputs, or maybe we need to do something in addition to that?

Eric K Richardson

@ekrich

@cedricmjohn Great that you put what you found here as well. If you are really motivated you can always put in a PR to improve the docs.

BISHNU SHANKAR

@bishnushankar95_twitter

Is there any way to run a python file using scalapy?

BISHNU SHANKAR

@bishnushankar95_twitter

Hey All,
I was testing parallel execution of scalapy with multiple instances of Module in parallel. My module uses CPython libraries like pandas and NumPy. I was reusing the module instance and using the applyDynamic method in each iteration. I noticed that there are over 10,000 files opened by the process. Is it expected behavior?

Eric K Richardson

@ekrich

@shadaj Have you seen this? https://docs.hpyproject.org/en/latest/index.html Check out handles.

Eddie Hatfield

@e-hat

hey guys -- i'm a student doing CS + math. I love scala + language runtimes + FP. i'm probably not much of a user of this project, but I would love to contribute and learn how real world scala projects are setup. i look forward to working with you all!

Eric K Richardson

@ekrich

Hey @e-hat - not to recruit you away from this project but there is also Scala Native if you like the bare metal stuff with C and garbage collector etc. https://github.com/scala-native/scala-native or Scala.js if you like the front end JavaScript stuff. https://github.com/scala-js/scala-js

shadaj

@shadaj:matrix.org

[m]

@cedricmjohn: hmm, that's interesting to see; I wonder if the JVM's built-in support for GUIs is interfering with Python somehow. In theory GUI apps should work as-is, could you try running the code with Scala Native and see if that works? That'll tell us whether it's a ScalaPy core issue or something specific to the JVM.

@bishnushankar95_twitter: even if you're loading the same module multiple times, Python should be caching the loaded modules, so my guess is that Pandas might be creating some temporary files locally? (though the number of files seems quite large even for that)

@ekrich: yes!! I've been watching HPy closely since it seems to offer a much nicer API for interactive with Python values. So far, though, HPy is more focused on implementing extensions so it's not clear how well it would work in the embedding case.

@e-hat: have you considered applying to Google Summer of Code (https://summerofcode.withgoogle.com/organizations/5449337515016192/)? ScalaPy, Scala Native, and Scala.js are all participating so you might find a project that aligns well with your interests there!

Eric K Richardson

@ekrich

@shadaj:matrix.org that is a great idea for @e-hat - there are lots of nice projects. I think the proposals are due very soon though.

@shadaj:matrix.org Denys pointed the project out here. :point_up: April 8, 2021 9:23 AM and thus we have scala-native/scala-native#2248

Eddie Hatfield

@e-hat

@shadaj:matrix.org @ekrich Thanks for linking those projects! Gotta love the friendly scala community. And I'll check out GSoC and maybe send an application in -- I am working full time this summer so IDK if I can handle GSoC on top of that, but we'll see. Can't hurt to apply.

Eric K Richardson

@ekrich

:thumbsup:

BISHNU SHANKAR

@bishnushankar95_twitter

I have a function def execute(a: String)(params:Any*): py.Dynamic = py.native
How to access the params in the python File. I am not able to understand how to use params. Can anyone help me with this? @shadaj:matrix.org

It will be helpful if the answer has a python function where I can access the params

scalavision

@scalavision

hello everyone, does it exist any scalapy wrapper for numpy, pytorch etc? Maybe there exists something with scala-native?

shadaj

@shadaj:matrix.org

[m]

@scalavision: at the moment, there aren't well-maintained static-type wrappers for those libraries (though this is something that will be improved soon with static type generation!); you can still use these libraries with the dynamically typed API in the meantime

Scala Native is more of a backend for ScalaPy, in terms of compatibility all Python libraries can be used the same way from either the JVM or Scala Native

1 reply

ScalaWilliam

@ScalaWilliam

Hi, how can you call 'del' in Scalapy?

shadaj

@shadaj:matrix.org

[m]

@ScalaWilliam: shadaj/scalapy#175 :)

ScalaWilliam

@ScalaWilliam

<3 :-) does it include 'del x' where x is a plain object not a dict? for what I'm doing I need to 'del' to delete a process (not my api!)

I did:
CPythonInterpreter.set("something", Writer.anyWriter[py.Any].write(something.as[py.Any]))
CPythonInterpreter.eval("delete something")

shadaj

@shadaj:matrix.org

[m]

ah, it doesn't include that; I'll have to think about how to implement that, since there isn't an equivalent Python API to that function, but the del method should be triggered by cleanup on a reference count decrement

it might be possible to offer an unsafe API on py.Any that lets you release a Python value even if it's still held by Scala code

ScalaWilliam

@ScalaWilliam

coool! :-)

shadaj

@shadaj:matrix.org

[m]

that would let you directly drop the reference instead of being at the mercy of the garbage collector

shadaj

@shadaj:matrix.org

[m]

@ScalaWilliam: just put up shadaj/scalapy#177 to resolve the missing APIs for your use-case!

ScalaWilliam

@ScalaWilliam

amazing @shadaj:matrix.org :-)

BISHNU SHANKAR

@bishnushankar95_twitter

I am trying to run scalapy from intellij in my window machine. I am using anaconda python with python version 3.8.5 . I am getting the below error
`java.lang.UnsatisfiedLinkError: Unable to load library 'python3.8m':
The specified module could not be found.

The specified module could not be found.

Native library (win32-x86-64/python3.8m.dll) not found in resource path `

shadaj

@shadaj:matrix.org

[m]

@bishnushankar95_twitter: did you make sure to activate the conda environment before launching SBT? (there can be some trickery required when using IDEs, which may launch SBT in the wrong environment); generally ScalaPy should work fine with conda, can you run python-config --ldflags to check where the libraries are installed?

BISHNU SHANKAR

@bishnushankar95_twitter

@shadaj:matrix.org I also tried the same by installing python on my machine. It is throwing the same exception. I tried python-config --ldflags but it is giving me python-config is not recognized as an internal or external command,operable program or batch file. error. Please help me with this

shadaj

@shadaj:matrix.org

[m]

@bishnushankar95_twitter: hmm, what platform are you running on and how did you install Python? I wonder if somehow you don't have the shared libraries

BISHNU SHANKAR

@bishnushankar95_twitter

I am running my scalapy program in a machine machine. I have downloaded python from the official website "https://www.python.org/downloads/release/python-385/"

Let me know the shared libraries that should be there before running scalapy program

@shadaj:matrix.org Please share the list of prerequisite libraries for scalapy

shadaj

@shadaj:matrix.org

[m]

@bishnushankar95_twitter: Hmm, can you try running

from distutils import sysconfig;
print(sysconfig.get_config_var("LIBDIR"))

in Python? that should tell you where the shared libraries are

kartikp4892

@kartikp4892

Is it possible to extend a python class/module using scalapy?

Something like class scala_class extends py_class

Second, I want to access python decorators on scala functions, is this possible with scalapy?

kartikp4892

@kartikp4892

So basically, I want to use base class and decorators from python and want to extend it in scala, the use python decorators on extended scala class functions.

BISHNU SHANKAR

@bishnushankar95_twitter

@shadaj:matrix.org I am getting None as the output of above code. Can you tell me what is the issue?

BISHNU SHANKAR

@bishnushankar95_twitter

@shadaj:matrix.org I have set the LIBDIR = '{pythonbase}/Lib/site-packages' but still i am getting the same error

shadaj

@shadaj:matrix.org

[m]

@kartikp4892: unfortunately you can't with high-level APIs right now, you should be able to create a subclass with https://docs.python.org/3/library/types.html

Decorators have a similar story, there's no native support in ScalaPy but you should be able to use Python APIs for them

@bishnushankar95_twitter: hmm, this makes it seem that you have a Python installation without the shared libraries; do you see any .so (linux) or .dylib (mac) files next to your Python installation?

BISHNU SHANKAR

@bishnushankar95_twitter

@shadaj:matrix.org I am using a window machine. There are many .ddl file in the python installation path

Nir Dunetz

@nirdunetz

Hi all, ive created python module and now I want to call it from scala using ScalaPy.
I started with call the module and the specific function:

 val testScalaPy = py.Module("my_module.test_feature")
 val res = testScalaPy.TestFeature.extract(payload = 1)

And its works perfectly.

Now, i would like to be able to pass case class to the extract function:

 case class Payload(value: Int) 
  val res = testScalaPy.TestFeature.extract(payload(100))

but im getting:

type mismatch;
found : com.riskified.poc.ScalaPyPocTest.Payload
required: me.shadaj.scalapy.py.Any
error after rewriting to feature.selectDynamic("TestFeature").applyDynamicNamed("extract")(scala.Tuple2("payload", Payload(200)))
possible cause: maybe a wrong Dynamic method signature?

Any ideas on how to implement writer/reader or if this ability exists already?

shadaj

@shadaj:matrix.org

[m]

@nirdunetz: unfortunately ScalaPy doesn't support converting case classes to Python right now; your best bet would be to create a dictionary and convert each field recursively (and maybe later convert this logic to be handled by a custom Writer derivation)

Where communities thrive

People

Repo info

Activity