Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 28 00:34
    scala-steward closed #325
  • Jan 28 00:34
    scala-steward commented #325
  • Jan 28 00:34
    scala-steward opened #329
  • Jan 22 15:57
    sbrunk commented #320
  • Jan 15 19:14
    scala-steward closed #286
  • Jan 15 19:14
    scala-steward commented #286
  • Jan 15 19:14
    scala-steward opened #328
  • Jan 09 08:54
    alexarchambault closed #309
  • Jan 09 08:54
    alexarchambault commented #309
  • Jan 08 02:10
    scala-steward opened #327
  • Nov 28 2022 20:50
    scala-steward opened #326
  • Nov 24 2022 20:10
    scala-steward closed #321
  • Nov 24 2022 20:10
    scala-steward commented #321
  • Nov 24 2022 20:10
    scala-steward opened #325
  • Nov 24 2022 20:10
    scala-steward closed #323
  • Nov 24 2022 20:10
    scala-steward commented #323
  • Nov 24 2022 20:10
    scala-steward opened #324
  • Nov 17 2022 20:16

    shadaj on v0.5.3

    (compare)

  • Nov 17 2022 03:09

    shadaj on main

    Release v0.5.3 (compare)

  • Nov 12 2022 21:20

    shadaj on main

    Add changelog item for #297 (compare)

Gabriel Claramunt
@gclaramunt
What type I need to use in Scala to get a Python's dict ?
Gabriel Claramunt
@gclaramunt
(for now, I'm using a list and convert it to dict on the python side, but I'm not totally happy changing the python side)
shadaj
@shadaj:matrix.org
[m]
@gclaramunt: you can read it as a mutable.Map[K, V]!
Gabriel Claramunt
@gclaramunt
nice... and how I pass a Map to python? mutable.Map too?
shadaj
@shadaj:matrix.org
[m]
for sending a map to Python you'll need an immutable map (because we don't currently support proxies from Python back into Scala for maps)
Gabriel Claramunt
@gclaramunt
how do I do the conversion? toPythonProxy ?
Gabriel Claramunt
@gclaramunt
(the library is awesome, btw)
shadaj
@shadaj:matrix.org
[m]
it will automatically happen! (since there's no choice between proxy and copy)
that's great to hear! please let us know if there are any things we can do to make it more awesome!
Gabriel Claramunt
@gclaramunt
Can I pass an arbitrary object created in Scala as a parameter to a python call? Couldn't find any example
(I might get away with just passing a Map, but anyway)
shadaj
@shadaj:matrix.org
[m]

Not arbitrary objects at the moment. There have been discussions about making this possible through some type of annotation eventually, but it will be some time before it's available.

In the meantime, yeah passing maps is probably the way to go.

Gabriel Claramunt
@gclaramunt
passing a map worked just fine
Keith Alcock
@kwalcock
What is the best way to execute a Python "script" with ScalaPy? I have a string in Scala containing Python code that includes some import statements, defines some functions, and then runs one of the functions, particularly if name == "main":. Right now I have to write the script to a file and then execute a Python process with the name of the file as an argument. This gets expensive if it has to be done repeatedly. I'd like to do something like py.eval() with more than an expression, so maybe py.exec(). Is there any way to do that? Thanks.
shadaj
@shadaj:matrix.org
[m]
you can just use the Python exec function via ScalaPy, as in py.Dynamic.global.exec(...)
Keith Alcock
@kwalcock

Thank you! IntelliJ wasn't showing that possibility. That brought is as far as "Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SystemError'> frame does not exist" for the test program

package org.clulab.pdf2txt.apps

import me.shadaj.scalapy.py

object PythonApp extends App {
{
val script = "3 + 4"
val pythonResult = py.eval(script)
val result = pythonResult.as[Int]
println(result)
}
{
val script = """print("Hello, world!")""" + "\n"
val pythonResult = py.Dynamic.global.exec(script)
println(pythonResult)
}
}

where eval() works, but exec() doesn't. The value for javaOptions seems to be reasonable: [debug] javaOptions: List(-Djna.library.path=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu:/usr/lib, -Dscalapy.python.library=python3.8, -Dscalapy.python.programname=/bin/python3)

2 replies
shadaj
@shadaj:matrix.org
[m]
hmm, yeah we should definitely get exec working; execManyLines is a reasonable fallback for now but we generally want to avoid people depending on CPythonInterpreter
Keith Alcock
@kwalcock
Does anyone have tips on how to deal with Python iterators?
import numpy as np
from tqdm.notebook import tqdm
n_examples = 10
epoch = 0
indices = np.arange(n_examples)
for i in tqdm(indices, desc=f'epoch {epoch+1}'):
1 reply
Keith Alcock
@kwalcock
Where is NumPy defined? On the main documentation page there is an example val np = py.module("numpy").as[NumPy] but the compiler does not know about NumPy. There is a scala-numpy project, but it is not clear which version of that is required or whether to add the library dependency in the main build.sbt file or somewhere under project. So far nothing has worked.
1 reply
shadaj
@shadaj:matrix.org
[m]
@kwalcock: we should probably have a reader for converting Python iterators into Scala iterators, but right now you'll have to create one manually by calling __next__
Keith Alcock
@kwalcock

Thanks. If I see

for value in thing:

it can be changed to

val iter = thing.__iter__()
while (true) {
val value = iter.__next__()
}

with the addition of a try and catch for when the PythonException of class StopIteration is thrown at the end.

Nachiket Deo
@NachiDeo_gitlab
Hello all, I want to use scalapy to use a class that I defined in python. I would like to know what would be correct syntax in order to this.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: first you'll have to import the module containing the class using py.module("modulename") and then you can construct the class using Python-like syntax module.MyClass(params)
2 replies
Eric K Richardson
@ekrich
Is ScalaPy working on Scala 3 and the latest Scala Native?
shadaj
@shadaj:matrix.org
[m]
It should work! We haven't updated the Scala Native version recently but there haven't been any binary-incompatible releases afaik.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: you may need to create an __init__.py file to instruct Python to treat the local directory as a module; otherwise, imports should follow the regular rules
Nachiket Deo
@NachiDeo_gitlab
@shadaj:matrix.org Thanks. I created the init.py file but I think I have some questions about workings of ScalaPy. So a Python interpreter would be created when Scala code containing scalapy constructs is executed. How would the interpreter understand that it has took look for custom designed Python class is located at? I created the init.py file but I think I would need to place the python file with class and init.py file in specific directory.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: so the Python interpreter automatically uses the current folder it is launched in to resolve modules, which is why creating additional files in the directory you're launching from should work
Sören Brunk
@sbrunk

Hi,
perhaps I'm missing something obvious, but I can't figure out how to combine facades with sequence conversions.
I have a Python function that takes a list

def f(s: list[int]) -> str: ...

Then on the Scala side I have a corresponding facade:

@py.native
class PyService extends py.Object {
  def f(s: Seq[Int]): String = py.native
}

Trying to compile it complains about the missing writer.

[error] 116 |    def f(s: Seq[Int]): String = py.native
[error]     |                                 ^^^^^^^^^
[error]     |There is no implicit for me.shadaj.scalapy.readwrite.Writer[scala.collection.immutable.Seq[scala.Int]]

So I looked at convertCopy from SeqConverters, to create a Writer, but it wraps everthing in py.Any so it's not directly usable.

Sören Brunk
@sbrunk
So after looking into the implementation of SeqConverters I was able to create a writer that seems to work:
implicit def seqWriter[T](implicit elemConvertable: ConvertableToSeqElem[T]): Writer[Seq[T]] = new Writer[Seq[T]] {
  override def write(seq: Seq[T]): PyValue = PyValue.fromNew(CPythonInterpreter.createListCopy(seq, elemConvertable.convertCopy))
}
But I'm still wondering if there's a better way to do it.
Sören Brunk
@sbrunk
Somewhat related, is it possible to copy a buffer, or byte array from Scala to Python in a performant way i.e. creating bytes or bytearray on the Python side?
1 reply
Gianluca Aguzzi
@cric96

Hey! :) first of all congrats for the library -- very interesting work :)
Am I just wondering: It is possible to define python classes in Scala? Consider for instance this example of PyTorch neural network definition:

## Simple network
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How can I reach a similar result through scalapy?

Keith Alcock
@kwalcock
How can one convert a Python list of integers that may contain a None? as[IndexedSeq[Int]] will throw an exception. I'm writing a new Reader[Int]. Is that the way to go? It looks like I need to avoid CPythonAPI.PyLong_AsLongLong(r) which seems to cause problems on the None.
Keith Alcock
@kwalcock

In

override def readNative(r: Platform.Pointer): Int

I could ask r for r.equals(<pointer of py.Dynamic.global.None>), however there doesn't seem to be a way to get to the pointer of None except in the debugger.

Keith Alcock
@kwalcock
For now it's just
    override def readNative(r: Platform.Pointer): Int = {
      val res = CPythonAPI.PyLong_AsLongLong(r)

      if (res == -1) {
        getAndClearError()
        none
      }
      else
        res.toInt
    }
shadaj
@shadaj:matrix.org
[m]
@cric96: not directly right now, it's something we'd like to add in the near future though; in the meantime you can create classes at runtime using type() (https://www.geeksforgeeks.org/create-classes-dynamically-in-python/)
@kwalcock: ah, the ideal way to do this would probably be to have a Reader[Option[T]] implementation that can check if the value is None and just recursively use the Reader[T] if it is not
that's something that would be welcome as a PR!
Gianluca Aguzzi
@cric96
Thank you so much shadaj! :) I try with type() then!
Alexandre Archambault
@alexarchambault
Hello here, does anyone know if there's a way to call a python method with varargs, like py.Dynamic.global.print(args0: _*), using the standard ScalaPy API. I'm getting a "applyDynamic does not support passing a vararg parameter" error.
3 replies
Maybe this needs to be added? (I'm calling directly CPythonInterpreter.callGlobal myself as a workaround right now…)
Gianluca Aguzzi
@cric96
@shadaj:matrix.org why with scala 3 macro isn't possible to use trait as a facade? There are technical limitations?
tomzheng77
@tomzheng77
hey guys, just want to say thanks for the awesome library. I've finished my setup just now and being able to call python from JVM/native is amazing
shadaj
@shadaj:matrix.org
[m]
@cric96: yeah, instantiating anonymous subclasses of traits is something that Scala 3 doesn't currently support in macros, which is why we are requiring classes for now
vonchav
@voonchav_gitlab
Hello, is there any plan to cut a new release? :)
shadaj
@shadaj:matrix.org
[m]
yes, probably in the next few days now that we got rich error messages merged in!
vonchav
@voonchav_gitlab
Fantastic! Thank you all for all the great work! I too look forward to the better error reporting. :)
vonchav
@voonchav_gitlab
Thanks for releasing 0.5.3! However, I don't see the new artifact on Maven. There is a new 0.5.2 snapshot though. :) The publish step of Github Action looks finished and successful though.
shadaj
@shadaj:matrix.org
[m]
need to publish the tag! that will trigger the full release