shadaj/scalapy

Andy Czerwonka

@andyczerwonka

Wondering if anyone can shed some light here. I'm trying to integrate ScalaPy into our environment. We use the DockerPlugin, so I have added Python to the environment like so:

Cmd("USER", "root")
Cmd("ENV", "PYTHONUNBUFFERED 1")
Cmd("RUN", "apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python")
Cmd("RUN", "python3 -m ensurepip")
Cmd("RUN", "pip3 install --no-cache --upgrade pip setuptools")

I then hit an endpoint that invokes a very simple piece of code:

  def sum(acl: AccessControlSession, xs: Seq[Double]): Future[Either[SessionError, SumAnswer]] = {
    import me.shadaj.scalapy.py
    import me.shadaj.scalapy.py.SeqConverters
    val pythonList = py.Dynamic.global.list(xs.toPythonProxy)
    val sum = py.Dynamic.global.sum(pythonList).as[Double]
    Future.successful(Right(SumAnswer(sum)))
  }

And Boom!!! I've killed the container.

| Fatal Python error: create_gil: PyCOND_INIT(gil->cond) failed
| Python runtime state: preinitialized
|
| Current thread 0x00007f43614a7640 (most recent call first):
| <no Python frame>
| #
| # A fatal error has been detected by the Java Runtime Environment:
| #
| #  SIGSEGV (0xb) at pc=0x00007f445cfe793b, pid=1, tid=224
| #
| # JRE version: OpenJDK Runtime Environment Temurin-11.0.13+8 (11.0.13+8) (build 11.0.13+8)
| # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (11.0.13+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
| # Problematic frame:
| # C  [libc.so.6+0x2693b]  abort+0x1ed
| #
| # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /opt/docker/core.1)
| #
| # An error report file with more information is saved as:
| # /tmp/hs_err_pid1.log
| #
| # If you would like to submit a bug report, please visit:
| #   https://github.com/adoptium/adoptium-support/issues
| # The crash happened outside the Java Virtual Machine in native code.
| # See problematic frame for where to report the bug.
| #

1 reply

Miguel Pérez Pasalodos

@mpasa

Hello! I'm trying to implement a writer and reader for Option[A] and not sure exactly how. I'll want to check for NoneType in Python. What I have right now for the writer is:

implicit def optionWriter[A](implicit aWriter: Writer[A], anyWriter: Writer[py.Any]): Writer[Option[A]] = new Writer[Option[A]] {
  override def write(opt: Option[A]) = {
    opt match {
      case Some(a) => aWriter.write(a)
      case None => anyWriter.write(py.eval("None"))
    }
  }
}

The py.eval part is a bit shady, and not sure if there is a better way. For the reader, not sure how to start checking if a Platform.Pointer or PyValue is None.

vonchav

@voonchav_gitlab

Hi all, I'm pointing SCALAPY_PYTHON_PROGRAMNAME to a virtualenv's python3 path (e.g. /Users/foo/.virtualenvs/scalapy/bin/python3). It's working if I do py.module("numpy") as I have numpy installed in the virtualenv. The directory of the actual virtualenv instance is at /Users/foo/python/venvs/scalapy.

Now, I want to load a Python script (foo.py) under a sub-directory test in that virtualenv directory. (The full path is /Users/foo/python/venvs/scalapy/test/foo.py) but I can't figure out how to load test.foo successfully. The py.module("test.foo") call is failing. How do I call py.module to load test.foo module? Thanks in advance!

shadaj

@shadaj:matrix.org

[m]

@mpasa: the Reader/Writer API is actually ScalaPy internal, and isn't intended to be implemented by end-users. If you want richer, higher-level conversions, you should probably define your own Reader/Writer traits that convert between Scala values and the higher-level py.Any subtypes.

We should probably document this / only allow ScalaPy internal code to implement these traits.

@voonchav_gitlab: have you set up an __init__.py file in the test folder? you'll need that so that Python recognizes it as a module

Miguel Pérez Pasalodos

@mpasa

@shadaj:matrix.org thanks! I have another question. Is it possible to run different interpreters in the same JVM? It doesn't look possible as CPythonInterpreter is an object

shadaj

@shadaj:matrix.org

[m]

@mpasa: unfortunately not, the story with subinterpreters with the native CPython APIs is quite a mess (https://docs.python.org/3/c-api/init.html#bugs-and-caveats) so in ScalaPy we chose to focus on just a single interpreter instance

however, ScalaPy does support safe multithreaded access to Python values

and if you're using a Python library backed by native code, you might be able to get good scaling if that native code appropriately releases the GIL

Gabriel Claramunt

@gclaramunt

What type I need to use in Scala to get a Python's dict ?

Gabriel Claramunt

@gclaramunt

(for now, I'm using a list and convert it to dict on the python side, but I'm not totally happy changing the python side)

shadaj

@shadaj:matrix.org

[m]

@gclaramunt: you can read it as a mutable.Map[K, V]!

Gabriel Claramunt

@gclaramunt

nice... and how I pass a Map to python? mutable.Map too?

shadaj

@shadaj:matrix.org

[m]

for sending a map to Python you'll need an immutable map (because we don't currently support proxies from Python back into Scala for maps)

Gabriel Claramunt

@gclaramunt

how do I do the conversion? toPythonProxy ?

Gabriel Claramunt

@gclaramunt

(the library is awesome, btw)

shadaj

@shadaj:matrix.org

[m]

it will automatically happen! (since there's no choice between proxy and copy)

that's great to hear! please let us know if there are any things we can do to make it more awesome!

Gabriel Claramunt

@gclaramunt

Can I pass an arbitrary object created in Scala as a parameter to a python call? Couldn't find any example

(I might get away with just passing a Map, but anyway)

shadaj

@shadaj:matrix.org

[m]

Not arbitrary objects at the moment. There have been discussions about making this possible through some type of annotation eventually, but it will be some time before it's available.

In the meantime, yeah passing maps is probably the way to go.

Gabriel Claramunt

@gclaramunt

passing a map worked just fine

Keith Alcock

@kwalcock

What is the best way to execute a Python "script" with ScalaPy? I have a string in Scala containing Python code that includes some import statements, defines some functions, and then runs one of the functions, particularly if name == "main":. Right now I have to write the script to a file and then execute a Python process with the name of the file as an argument. This gets expensive if it has to be done repeatedly. I'd like to do something like py.eval() with more than an expression, so maybe py.exec(). Is there any way to do that? Thanks.

shadaj

@shadaj:matrix.org

[m]

you can just use the Python exec function via ScalaPy, as in py.Dynamic.global.exec(...)

Keith Alcock

@kwalcock

Thank you! IntelliJ wasn't showing that possibility. That brought is as far as "Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SystemError'> frame does not exist" for the test program

package org.clulab.pdf2txt.apps

import me.shadaj.scalapy.py

object PythonApp extends App {
{
val script = "3 + 4"
val pythonResult = py.eval(script)
val result = pythonResult.as[Int]
println(result)
}
{
val script = """print("Hello, world!")""" + "\n"
val pythonResult = py.Dynamic.global.exec(script)
println(pythonResult)
}
}

where eval() works, but exec() doesn't. The value for javaOptions seems to be reasonable: [debug] javaOptions: List(-Djna.library.path=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu:/usr/lib, -Dscalapy.python.library=python3.8, -Dscalapy.python.programname=/bin/python3)

2 replies

shadaj

@shadaj:matrix.org

[m]

hmm, yeah we should definitely get exec working; execManyLines is a reasonable fallback for now but we generally want to avoid people depending on CPythonInterpreter

Keith Alcock

@kwalcock

Does anyone have tips on how to deal with Python iterators?

import numpy as np
from tqdm.notebook import tqdm
n_examples = 10
epoch = 0
indices = np.arange(n_examples)
for i in tqdm(indices, desc=f'epoch {epoch+1}'):

1 reply

Keith Alcock

@kwalcock

Where is NumPy defined? On the main documentation page there is an example val np = py.module("numpy").as[NumPy] but the compiler does not know about NumPy. There is a scala-numpy project, but it is not clear which version of that is required or whether to add the library dependency in the main build.sbt file or somewhere under project. So far nothing has worked.

1 reply

shadaj

@shadaj:matrix.org

[m]

@kwalcock: we should probably have a reader for converting Python iterators into Scala iterators, but right now you'll have to create one manually by calling __next__

Keith Alcock

@kwalcock

Thanks. If I see

for value in thing:

it can be changed to

val iter = thing.__iter__()
while (true) {
val value = iter.__next__()
}

with the addition of a try and catch for when the PythonException of class StopIteration is thrown at the end.

Nachiket Deo

@NachiDeo_gitlab

Hello all, I want to use scalapy to use a class that I defined in python. I would like to know what would be correct syntax in order to this.

shadaj

@shadaj:matrix.org

[m]

@NachiDeo_gitlab: first you'll have to import the module containing the class using py.module("modulename") and then you can construct the class using Python-like syntax module.MyClass(params)

2 replies

Eric K Richardson

@ekrich

Is ScalaPy working on Scala 3 and the latest Scala Native?

shadaj

@shadaj:matrix.org

[m]

It should work! We haven't updated the Scala Native version recently but there haven't been any binary-incompatible releases afaik.

shadaj

@shadaj:matrix.org

[m]

@NachiDeo_gitlab: you may need to create an __init__.py file to instruct Python to treat the local directory as a module; otherwise, imports should follow the regular rules

Nachiket Deo

@NachiDeo_gitlab

@shadaj:matrix.org Thanks. I created the init.py file but I think I have some questions about workings of ScalaPy. So a Python interpreter would be created when Scala code containing scalapy constructs is executed. How would the interpreter understand that it has took look for custom designed Python class is located at? I created the init.py file but I think I would need to place the python file with class and init.py file in specific directory.

shadaj

@shadaj:matrix.org

[m]

@NachiDeo_gitlab: so the Python interpreter automatically uses the current folder it is launched in to resolve modules, which is why creating additional files in the directory you're launching from should work

Sören Brunk

@sbrunk

Hi,
perhaps I'm missing something obvious, but I can't figure out how to combine facades with sequence conversions.
I have a Python function that takes a list

def f(s: list[int]) -> str: ...

Then on the Scala side I have a corresponding facade:

@py.native
class PyService extends py.Object {
  def f(s: Seq[Int]): String = py.native
}

Trying to compile it complains about the missing writer.

[error] 116 |    def f(s: Seq[Int]): String = py.native
[error]     |                                 ^^^^^^^^^
[error]     |There is no implicit for me.shadaj.scalapy.readwrite.Writer[scala.collection.immutable.Seq[scala.Int]]

So I looked at convertCopy from SeqConverters, to create a Writer, but it wraps everthing in py.Any so it's not directly usable.

Sören Brunk

@sbrunk

So after looking into the implementation of SeqConverters I was able to create a writer that seems to work:

implicit def seqWriter[T](implicit elemConvertable: ConvertableToSeqElem[T]): Writer[Seq[T]] = new Writer[Seq[T]] {
  override def write(seq: Seq[T]): PyValue = PyValue.fromNew(CPythonInterpreter.createListCopy(seq, elemConvertable.convertCopy))
}

But I'm still wondering if there's a better way to do it.

Sören Brunk

@sbrunk

Somewhat related, is it possible to copy a buffer, or byte array from Scala to Python in a performant way i.e. creating bytes or bytearray on the Python side?

1 reply

Gianluca Aguzzi

@cric96

Hey! :) first of all congrats for the library -- very interesting work :)
Am I just wondering: It is possible to define python classes in Scala? Consider for instance this example of PyTorch neural network definition:

## Simple network
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How can I reach a similar result through scalapy?

Keith Alcock

@kwalcock

How can one convert a Python list of integers that may contain a None? as[IndexedSeq[Int]] will throw an exception. I'm writing a new Reader[Int]. Is that the way to go? It looks like I need to avoid CPythonAPI.PyLong_AsLongLong(r) which seems to cause problems on the None.

Keith Alcock

@kwalcock

override def readNative(r: Platform.Pointer): Int

I could ask r for r.equals(<pointer of py.Dynamic.global.None>), however there doesn't seem to be a way to get to the pointer of None except in the debugger.

Keith Alcock

@kwalcock

For now it's just

    override def readNative(r: Platform.Pointer): Int = {
      val res = CPythonAPI.PyLong_AsLongLong(r)

      if (res == -1) {
        getAndClearError()
        none
      }
      else
        res.toInt
    }

shadaj

@shadaj:matrix.org

[m]

@cric96: not directly right now, it's something we'd like to add in the near future though; in the meantime you can create classes at runtime using type() (https://www.geeksforgeeks.org/create-classes-dynamically-in-python/)

@kwalcock: ah, the ideal way to do this would probably be to have a Reader[Option[T]] implementation that can check if the value is None and just recursively use the Reader[T] if it is not

that's something that would be welcome as a PR!

Gianluca Aguzzi

@cric96

Thank you so much shadaj! :) I try with type() then!

Where communities thrive

People

Repo info

Activity