Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 23 21:42
    dependabot[bot] labeled #236
  • Nov 23 21:42
    dependabot[bot] opened #236
  • Nov 23 21:42

    dependabot[bot] on npm_and_yarn

    Bump algoliasearch-helper from … (compare)

  • Nov 09 14:10
    scala-steward opened #235
  • Nov 08 21:36
    scala-steward opened #234
  • Nov 03 19:48

    shadaj on v0.5.1

    (compare)

  • Nov 03 19:33

    shadaj on main

    Release v0.5.1 (compare)

  • Nov 02 02:29
    scala-steward opened #233
  • Oct 27 14:59
    Tinawyn edited #232
  • Oct 27 14:59
    Tinawyn edited #232
  • Oct 27 14:59
    Tinawyn edited #232
  • Oct 27 14:57
    Tinawyn opened #232
  • Oct 23 21:47

    shadaj on py-local-docs

    (compare)

  • Oct 23 21:47

    shadaj on main

    Add docs explaining how to use … (compare)

  • Oct 23 21:47
    shadaj closed #231
  • Oct 23 21:47
    shadaj closed #189
  • Oct 23 20:41
    shadaj opened #231
  • Oct 23 20:41
    shadaj milestoned #231
  • Oct 23 20:41

    shadaj on py-local-docs

    Add docs explaining how to use … (compare)

  • Oct 23 19:56
    shadaj milestoned #141
kartikp4892
@kartikp4892
Second, I want to access python decorators on scala functions, is this possible with scalapy?
So basically, I want to use base class and decorators from python and want to extend it in scala, the use python decorators on extended scala class functions.
BISHNU SHANKAR
@bishnushankar95_twitter
@shadaj:matrix.org I am getting None as the output of above code. Can you tell me what is the issue?
BISHNU SHANKAR
@bishnushankar95_twitter
@shadaj:matrix.org I have set the LIBDIR = '{pythonbase}/Lib/site-packages' but still i am getting the same error
shadaj
@shadaj:matrix.org
[m]
@kartikp4892: unfortunately you can't with high-level APIs right now, you should be able to create a subclass with https://docs.python.org/3/library/types.html
Decorators have a similar story, there's no native support in ScalaPy but you should be able to use Python APIs for them
@bishnushankar95_twitter: hmm, this makes it seem that you have a Python installation without the shared libraries; do you see any .so (linux) or .dylib (mac) files next to your Python installation?
BISHNU SHANKAR
@bishnushankar95_twitter
@shadaj:matrix.org I am using a window machine. There are many .ddl file in the python installation path
Nir Dunetz
@nirdunetz

Hi all, ive created python module and now I want to call it from scala using ScalaPy.
I started with call the module and the specific function:

 val testScalaPy = py.Module("my_module.test_feature")
 val res = testScalaPy.TestFeature.extract(payload = 1)

And its works perfectly.

Now, i would like to be able to pass case class to the extract function:

 case class Payload(value: Int) 
  val res = testScalaPy.TestFeature.extract(payload(100)) 

but im getting:

type mismatch;
found : com.riskified.poc.ScalaPyPocTest.Payload
required: me.shadaj.scalapy.py.Any
error after rewriting to feature.selectDynamic("TestFeature").applyDynamicNamed("extract")(scala.Tuple2("payload", Payload(200)))
possible cause: maybe a wrong Dynamic method signature?

Any ideas on how to implement writer/reader or if this ability exists already?

shadaj
@shadaj:matrix.org
[m]
@nirdunetz: unfortunately ScalaPy doesn't support converting case classes to Python right now; your best bet would be to create a dictionary and convert each field recursively (and maybe later convert this logic to be handled by a custom Writer derivation)
BISHNU SHANKAR
@bishnushankar95_twitter
@shadaj:matrix.org Any solutions on the issue i am facing?
Nir Dunetz
@nirdunetz
@shadaj:matrix.org Thanks for the quick response.
1.There is a plan to support case class conversation?
2.Wdym convert the logic to Writer? Writer that convert to map? Can you show an example how to convert it to map and than use the writer?
let's say I have:
 case class Payload(total: Int,  info: Info)
 case class Info(firstName: Name, lastName: String)    
 case class Name(nameVal: String)
shadaj
@shadaj:matrix.org
[m]

@nirdunetz: not in the near future, since unlike JavaScript it's not immediately clear exactly what a case class should map to in Python (an object? a dictionary?)

you'll need to use something like https://github.com/softwaremill/magnolia/tree/scala2, which can extract out info on which fields are in each case class and what are their types, and then write code using ScalaPy APIs to construct a Python dictionary (or whatever type you want) and add the fields to it by recursively converting them

basically, you can imagine putting together a writer for one of your classes as
def write(v: Payload): py.Object = {
  val out = py.global.dict()
  out.bracketUpdate("total", implicitly[Writer[Int]].write(out.total)
  ...
  out
the idea is that Magnolia lets you abstract things so that you don't have to write this by hand for every type
Nir Dunetz
@nirdunetz
@shadaj:matrix.org Many thanks!
limar63
@limar63
Hi! I am a bit rusty and just starting with getting comfortable with sbt/projects etc. Can someone tell what is the reason for error Native library (win32-x86-64/python3.dll) not found in resource path ? It happened when I tried to println the example given on the starting page of ScalaPy
val listLengthPython = py.Dynamic.global.len(List(1, 2, 3).toPythonProxy)
println(listLengthPython)
and looks like it's implying I have no Python installed, but I just opened Python 3.7 32 bit version. It's on Windows (you could guess it because DLL). Can someone help with realizing what I am doing wrong?
shadaj
@shadaj:matrix.org
[m]
@limar63: you may want to try the (experimental) setup in shadaj/scalapy#210, which is designed to make finding the native libraries a lot easier
3 replies
Eric K Richardson
@ekrich
@shadaj:matrix.org Most of the Scala world has moved to discord - there were some discussion about ScalaPy there today. https://discord.gg/9G7Cch4q Not sure if you want to setup a server for your projects.
scalavision
@scalavision
@shadaj:matrix.org , will there be a new release soon with support for scala 3? Or is it still a lot of more work to do?
Alexey Novakov
@novakov-alexey
Hi there. Perhaps I have weird question, but is it possible to call Scala code from Python? :smile:
shadaj
@shadaj:matrix.org
[m]
@scalavision will probably cut a release in the next week! although the latest commits are already on Maven Central
1 reply
@novakov-alexey yes! you can pass Scala functions into Python
Alexey Novakov
@novakov-alexey
@shadaj:matrix.org is there some example?
and how that would work? Python VM would start JVM and pass some Scala code snippet to execute?
shadaj
@shadaj:matrix.org
[m]
@novakov-alexey: see https://github.com/shadaj/scalapy/blob/main/core/shared/src/test/scala/me/shadaj/scalapy/py/LambdaTest.scala; ScalaPy works by embedding Python inside your Scala app, so the Scala code would have to initiate things
Alexey Novakov
@novakov-alexey
@shadaj:matrix.org thanks. I have a use-case where my Python application would call Scala program. Scala program then returns some Python Object for further processing by the Python application :-)
I would looking for an example from *.py file perspective
shadaj
@shadaj:matrix.org
[m]
@novakov-alexey: ah, that's not something ScalaPy supports; the focus is on using Python APIs from Scala, not the other way around. Perhaps Jep could work for your use case?
Alexey Novakov
@novakov-alexey
@shadaj:matrix.org I see, thanks for JEP direction, it is closer to what I would need but not ideal
Joesan
@joesan
Any ideas on how to get rid of the following error:
[error] /home/joesan/Projects/Private/ml-projects/object-classifier/src/main/scala/com/bigelectrons/animalclassifier/ImageLoader.scala:10:21: could not find implicit value for parameter jep: jep.Jep
[error] val np = py.module("numpy").as[NumPy]
I have the following as imports:
"me.shadaj" %% "scalapy-numpy" % "0.1.0",
"me.shadaj" %% "scalapy" % "0.2.0",
"me.shadaj" %% "scalapy-core" % "0.5.0",
Joesan
@joesan
@kamilkloch You had a similiar issue in the past. Would be interested to know how you solved it!
Joesan
@joesan
Looks like I have to use "me.shadaj" %% "scalapy-numpy" % "0.1.0+6-14ca0424"
but then I hit two other errors that says:
[error] Make sure that type Writer is in your classpath and check for conflicting dependencies with -Ylog-classpath.
[error] A full rebuild may help if 'NumPy.class' was compiled against an incompatible version of me.shadaj.scalapy.py.
[error] np.asarray(preProcessedImagesWithLabels)
[error] /home/joesan/Projects/Private/ml-projects/object-classifier/src/main/scala/com/bigelectrons/animalclassifier/ImageLoader.scala:22:17: could not find implicit value for parameter writer: me.shadaj.scalapy.py.Writer[org.bytedeco.opencv.opencv_core.Mat]
[error] np.asarray(preProcessedImagesWithLabels)
[error] ^
shadaj
@shadaj:matrix.org
[m]
scalapy-numpy is unfortunately quite a bit out of date, but there is active work that will bring back static typing for NumPy and TensorFlow soon! in the meantime, you'll have to use the dynamically-typed APIs or define your own facades
Joesan
@joesan
@shadaj:matrix.org Thanks for the reply. Could you point me to some examples?
bogorman
@bogorman_twitter

Hi. I know the facadeGen is alpha but i just tried to run it and i get this error. Any suggestion on how to get it running? I just want to generate some facades for some modules and play with it. I am only trying to generate it for the "builtins" module at the moment while testing. I can see in your scalacon-live folder it looks like the facadeGen did work for at that point in time. Thanks.

[info] running (fork) me.shadaj.scalapy.facadegen.Main
[error] Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'TypeError'> list object expected; got SequenceProxy
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$2(CPythonInterpreter.scala:328)
[error] at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:314)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)

bogorman
@bogorman_twitter
Changed the use of toPythonProxy to toPythonCopy and it seems to run.
Lorenzo Gabriele
@lolgab
Hi @shadaj:matrix.org,
I was playing with Scalapy in Scala Native and I whatever I do I leak memory ( unless I use py.local).
I wanted to ask you why the automatic free of memory can't be done in Scala Native.
Is it because Scala Native doesn't support finalize? Because it is deprecated in Java 9, maybe people continue to use it either way?
If it is because you need to call malloc there is a "better" ( hackish ) way to deal with that and have GC memory managed..
You can allocate an Array[Byte]of a certain dimension. And then with ByteArray.at(0) you get the pointer to the start of the arrays' data section. When the original Array is GC collected, you have successfully freed the memory. So you don't have to use malloc.
import scala.scalanative.runtime.ByteArray

val arr = ByteArray.alloc(size)
arr.at(0) // is your pointer
return arr // you don't want to lose the reference to the `ByteArray`object ahead of time
Scalapy could be the thing that would allow us in Scala Native to have a proper ecosystem of utility libraries to get from, while still use pure Scala for the main programs. It would be great if it worked out of the box with memory leaks so you can call Python libraries and forget about it.
I'm particularly interested in the cloud native ecosystem which is really thorough in Python!
Lorenzo Gabriele
@lolgab
*without memory leaks
Eric K Richardson
@ekrich
@lolgab Could you explain a bit more you take on ScalaPy and how this fits into Scala Native and Cloud Native Foundation tools?
Lorenzo Gabriele
@lolgab
@ekrich This answer from Odersky + the 2 replies explain very well why Scala Native would play nicely with Python: https://contributors.scala-lang.org/t/scala-native-next-steps/4216/75
About Cloud Native tools, nothing special about Python there.. It is just that Cloud Native is the area I'm personally interested in and official client libraries for famous clouds like AWS are only for few languages: C++, Go, Java, Javascript, .NET, Node.js, PHP, Python and Ruby.
Scala Native can use C++ libraries if you write C++ glue code with extern "C"functions, but it is a bit of work and if the Python library is fast enough and it's not what makes you slow, I don't see the reason to go that route.
Other dynamic languages like PHP, Ruby etc. can be probably integrated with a Scalapy like library, but I think Python ecosystem is bigger.. And we already have scalapy!
Eric K Richardson
@ekrich
@lolgab I see thanks, that make total sense.