Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 24 20:10
    scala-steward closed #321
  • Nov 24 20:10
    scala-steward commented #321
  • Nov 24 20:10
    scala-steward opened #325
  • Nov 24 20:10
    scala-steward closed #323
  • Nov 24 20:10
    scala-steward commented #323
  • Nov 24 20:10
    scala-steward opened #324
  • Nov 17 20:16

    shadaj on v0.5.3

    (compare)

  • Nov 17 03:09

    shadaj on main

    Release v0.5.3 (compare)

  • Nov 12 21:20

    shadaj on main

    Add changelog item for #297 (compare)

  • Nov 12 21:19
    shadaj milestoned #295
  • Nov 12 21:19
    shadaj milestoned #295
  • Nov 12 21:19
    shadaj milestoned #299
  • Nov 12 21:19
    shadaj milestoned #299
  • Nov 12 21:18
    shadaj milestoned #297
  • Nov 12 21:18
    shadaj milestoned #297
  • Nov 12 21:18
    shadaj closed #141
  • Nov 12 21:18
    shadaj commented #141
  • Nov 12 21:17
    shadaj demilestoned #141
  • Nov 12 21:17
    shadaj milestoned #141
  • Nov 12 21:17
    shadaj milestoned #243
shadaj
@shadaj:matrix.org
[m]
note that you won't be able to write multi-line Python code, so you may need to use the Python eval function inside to execute a larger block
Miguel Pérez Pasalodos
@mpasa

Thank you! It seems PyQuote or eval can't deal with assignations. I get the following:

 Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SyntaxError'> ('invalid syntax', ('<string>', 1, 2, 'f=spy_o_0'))

But I made it work like this:

import me.shadaj.scalapy.py
import py.PyQuote

def toPyValue[T](x: T)(implicit writer: Writer[T]): PyValue = writer.write(x)
def f(a: Int): Int = a*2
CPythonInterpreter.set("f",  toPyValue(f _))
println(py"f(5)")
shadaj
@shadaj:matrix.org
[m]
ah yeah, you won't be able to perform assignments with PyQuote since we send the Python snippet in expression mode (having an API that supports statement mode is something that should probably be implemented though)
vonchav
@voonchav_gitlab

Hi all, has anyone used/worked with pandas via ScalaPy? I tried to load a dataframe but got this error:

me.shadaj.scalapy.py.PythonException: <class 'TypeError'> 'DataFrame' object is not callable.

I'm able to import pandas as a module and print its version. It fails when I tried calling pd.DataFrame.from_records. The same code in Python is working in Python REPL.

val numpy = py.module("numpy")
val data = numpy.array(Seq( (1, "a"), (2, "b") ).toPythonCopy)

val pandas = py.module("pandas")

val pdVersion = pandas.__version__
println(s"Pandas version is $pdVersion")

// The following line fails
val dataframe = pandas.DataFrame.from_records(data)
vonchav
@voonchav_gitlab
^ Actually this info is not incomplete/incorrect. Sorry. I found what is causing this. It wasn't ScalaPy. :)
Andy Czerwonka
@andyczerwonka

Wondering if anyone can shed some light here. I'm trying to integrate ScalaPy into our environment. We use the DockerPlugin, so I have added Python to the environment like so:

Cmd("USER", "root")
Cmd("ENV", "PYTHONUNBUFFERED 1")
Cmd("RUN", "apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python")
Cmd("RUN", "python3 -m ensurepip")
Cmd("RUN", "pip3 install --no-cache --upgrade pip setuptools")

I then hit an endpoint that invokes a very simple piece of code:

  def sum(acl: AccessControlSession, xs: Seq[Double]): Future[Either[SessionError, SumAnswer]] = {
    import me.shadaj.scalapy.py
    import me.shadaj.scalapy.py.SeqConverters
    val pythonList = py.Dynamic.global.list(xs.toPythonProxy)
    val sum = py.Dynamic.global.sum(pythonList).as[Double]
    Future.successful(Right(SumAnswer(sum)))
  }

And Boom!!! I've killed the container.

| Fatal Python error: create_gil: PyCOND_INIT(gil->cond) failed
| Python runtime state: preinitialized
|
| Current thread 0x00007f43614a7640 (most recent call first):
| <no Python frame>
| #
| # A fatal error has been detected by the Java Runtime Environment:
| #
| #  SIGSEGV (0xb) at pc=0x00007f445cfe793b, pid=1, tid=224
| #
| # JRE version: OpenJDK Runtime Environment Temurin-11.0.13+8 (11.0.13+8) (build 11.0.13+8)
| # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (11.0.13+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
| # Problematic frame:
| # C  [libc.so.6+0x2693b]  abort+0x1ed
| #
| # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /opt/docker/core.1)
| #
| # An error report file with more information is saved as:
| # /tmp/hs_err_pid1.log
| #
| # If you would like to submit a bug report, please visit:
| #   https://github.com/adoptium/adoptium-support/issues
| # The crash happened outside the Java Virtual Machine in native code.
| # See problematic frame for where to report the bug.
| #
1 reply
Miguel Pérez Pasalodos
@mpasa

Hello! I'm trying to implement a writer and reader for Option[A] and not sure exactly how. I'll want to check for NoneType in Python. What I have right now for the writer is:

implicit def optionWriter[A](implicit aWriter: Writer[A], anyWriter: Writer[py.Any]): Writer[Option[A]] = new Writer[Option[A]] {
  override def write(opt: Option[A]) = {
    opt match {
      case Some(a) => aWriter.write(a)
      case None => anyWriter.write(py.eval("None"))
    }
  }
}

The py.eval part is a bit shady, and not sure if there is a better way. For the reader, not sure how to start checking if a Platform.Pointer or PyValue is None.

vonchav
@voonchav_gitlab

Hi all, I'm pointing SCALAPY_PYTHON_PROGRAMNAME to a virtualenv's python3 path (e.g. /Users/foo/.virtualenvs/scalapy/bin/python3). It's working if I do py.module("numpy") as I have numpy installed in the virtualenv. The directory of the actual virtualenv instance is at /Users/foo/python/venvs/scalapy.

Now, I want to load a Python script (foo.py) under a sub-directory test in that virtualenv directory. (The full path is /Users/foo/python/venvs/scalapy/test/foo.py) but I can't figure out how to load test.foo successfully. The py.module("test.foo") call is failing. How do I call py.module to load test.foo module? Thanks in advance!

shadaj
@shadaj:matrix.org
[m]
@mpasa: the Reader/Writer API is actually ScalaPy internal, and isn't intended to be implemented by end-users. If you want richer, higher-level conversions, you should probably define your own Reader/Writer traits that convert between Scala values and the higher-level py.Any subtypes.
We should probably document this / only allow ScalaPy internal code to implement these traits.
@voonchav_gitlab: have you set up an __init__.py file in the test folder? you'll need that so that Python recognizes it as a module
Miguel Pérez Pasalodos
@mpasa
@shadaj:matrix.org thanks! I have another question. Is it possible to run different interpreters in the same JVM? It doesn't look possible as CPythonInterpreter is an object
shadaj
@shadaj:matrix.org
[m]
@mpasa: unfortunately not, the story with subinterpreters with the native CPython APIs is quite a mess (https://docs.python.org/3/c-api/init.html#bugs-and-caveats) so in ScalaPy we chose to focus on just a single interpreter instance
however, ScalaPy does support safe multithreaded access to Python values
and if you're using a Python library backed by native code, you might be able to get good scaling if that native code appropriately releases the GIL
Gabriel Claramunt
@gclaramunt
What type I need to use in Scala to get a Python's dict ?
Gabriel Claramunt
@gclaramunt
(for now, I'm using a list and convert it to dict on the python side, but I'm not totally happy changing the python side)
shadaj
@shadaj:matrix.org
[m]
@gclaramunt: you can read it as a mutable.Map[K, V]!
Gabriel Claramunt
@gclaramunt
nice... and how I pass a Map to python? mutable.Map too?
shadaj
@shadaj:matrix.org
[m]
for sending a map to Python you'll need an immutable map (because we don't currently support proxies from Python back into Scala for maps)
Gabriel Claramunt
@gclaramunt
how do I do the conversion? toPythonProxy ?
Gabriel Claramunt
@gclaramunt
(the library is awesome, btw)
shadaj
@shadaj:matrix.org
[m]
it will automatically happen! (since there's no choice between proxy and copy)
that's great to hear! please let us know if there are any things we can do to make it more awesome!
Gabriel Claramunt
@gclaramunt
Can I pass an arbitrary object created in Scala as a parameter to a python call? Couldn't find any example
(I might get away with just passing a Map, but anyway)
shadaj
@shadaj:matrix.org
[m]

Not arbitrary objects at the moment. There have been discussions about making this possible through some type of annotation eventually, but it will be some time before it's available.

In the meantime, yeah passing maps is probably the way to go.

Gabriel Claramunt
@gclaramunt
passing a map worked just fine
Keith Alcock
@kwalcock
What is the best way to execute a Python "script" with ScalaPy? I have a string in Scala containing Python code that includes some import statements, defines some functions, and then runs one of the functions, particularly if name == "main":. Right now I have to write the script to a file and then execute a Python process with the name of the file as an argument. This gets expensive if it has to be done repeatedly. I'd like to do something like py.eval() with more than an expression, so maybe py.exec(). Is there any way to do that? Thanks.
shadaj
@shadaj:matrix.org
[m]
you can just use the Python exec function via ScalaPy, as in py.Dynamic.global.exec(...)
Keith Alcock
@kwalcock

Thank you! IntelliJ wasn't showing that possibility. That brought is as far as "Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SystemError'> frame does not exist" for the test program

package org.clulab.pdf2txt.apps

import me.shadaj.scalapy.py

object PythonApp extends App {
{
val script = "3 + 4"
val pythonResult = py.eval(script)
val result = pythonResult.as[Int]
println(result)
}
{
val script = """print("Hello, world!")""" + "\n"
val pythonResult = py.Dynamic.global.exec(script)
println(pythonResult)
}
}

where eval() works, but exec() doesn't. The value for javaOptions seems to be reasonable: [debug] javaOptions: List(-Djna.library.path=/usr/lib/python3.8/config-3.8-x86_64-linux-gnu:/usr/lib, -Dscalapy.python.library=python3.8, -Dscalapy.python.programname=/bin/python3)

2 replies
shadaj
@shadaj:matrix.org
[m]
hmm, yeah we should definitely get exec working; execManyLines is a reasonable fallback for now but we generally want to avoid people depending on CPythonInterpreter
Keith Alcock
@kwalcock
Does anyone have tips on how to deal with Python iterators?
import numpy as np
from tqdm.notebook import tqdm
n_examples = 10
epoch = 0
indices = np.arange(n_examples)
for i in tqdm(indices, desc=f'epoch {epoch+1}'):
1 reply
Keith Alcock
@kwalcock
Where is NumPy defined? On the main documentation page there is an example val np = py.module("numpy").as[NumPy] but the compiler does not know about NumPy. There is a scala-numpy project, but it is not clear which version of that is required or whether to add the library dependency in the main build.sbt file or somewhere under project. So far nothing has worked.
1 reply
shadaj
@shadaj:matrix.org
[m]
@kwalcock: we should probably have a reader for converting Python iterators into Scala iterators, but right now you'll have to create one manually by calling __next__
Keith Alcock
@kwalcock

Thanks. If I see

for value in thing:

it can be changed to

val iter = thing.__iter__()
while (true) {
val value = iter.__next__()
}

with the addition of a try and catch for when the PythonException of class StopIteration is thrown at the end.

Nachiket Deo
@NachiDeo_gitlab
Hello all, I want to use scalapy to use a class that I defined in python. I would like to know what would be correct syntax in order to this.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: first you'll have to import the module containing the class using py.module("modulename") and then you can construct the class using Python-like syntax module.MyClass(params)
2 replies
Eric K Richardson
@ekrich
Is ScalaPy working on Scala 3 and the latest Scala Native?
shadaj
@shadaj:matrix.org
[m]
It should work! We haven't updated the Scala Native version recently but there haven't been any binary-incompatible releases afaik.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: you may need to create an __init__.py file to instruct Python to treat the local directory as a module; otherwise, imports should follow the regular rules
Nachiket Deo
@NachiDeo_gitlab
@shadaj:matrix.org Thanks. I created the init.py file but I think I have some questions about workings of ScalaPy. So a Python interpreter would be created when Scala code containing scalapy constructs is executed. How would the interpreter understand that it has took look for custom designed Python class is located at? I created the init.py file but I think I would need to place the python file with class and init.py file in specific directory.
shadaj
@shadaj:matrix.org
[m]
@NachiDeo_gitlab: so the Python interpreter automatically uses the current folder it is launched in to resolve modules, which is why creating additional files in the directory you're launching from should work
Sören Brunk
@sbrunk

Hi,
perhaps I'm missing something obvious, but I can't figure out how to combine facades with sequence conversions.
I have a Python function that takes a list

def f(s: list[int]) -> str: ...

Then on the Scala side I have a corresponding facade:

@py.native
class PyService extends py.Object {
  def f(s: Seq[Int]): String = py.native
}

Trying to compile it complains about the missing writer.

[error] 116 |    def f(s: Seq[Int]): String = py.native
[error]     |                                 ^^^^^^^^^
[error]     |There is no implicit for me.shadaj.scalapy.readwrite.Writer[scala.collection.immutable.Seq[scala.Int]]

So I looked at convertCopy from SeqConverters, to create a Writer, but it wraps everthing in py.Any so it's not directly usable.

Sören Brunk
@sbrunk
So after looking into the implementation of SeqConverters I was able to create a writer that seems to work:
implicit def seqWriter[T](implicit elemConvertable: ConvertableToSeqElem[T]): Writer[Seq[T]] = new Writer[Seq[T]] {
  override def write(seq: Seq[T]): PyValue = PyValue.fromNew(CPythonInterpreter.createListCopy(seq, elemConvertable.convertCopy))
}
But I'm still wondering if there's a better way to do it.
Sören Brunk
@sbrunk
Somewhat related, is it possible to copy a buffer, or byte array from Scala to Python in a performant way i.e. creating bytes or bytearray on the Python side?
1 reply
Gianluca Aguzzi
@cric96

Hey! :) first of all congrats for the library -- very interesting work :)
Am I just wondering: It is possible to define python classes in Scala? Consider for instance this example of PyTorch neural network definition:

## Simple network
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How can I reach a similar result through scalapy?

Keith Alcock
@kwalcock
How can one convert a Python list of integers that may contain a None? as[IndexedSeq[Int]] will throw an exception. I'm writing a new Reader[Int]. Is that the way to go? It looks like I need to avoid CPythonAPI.PyLong_AsLongLong(r) which seems to cause problems on the None.
Keith Alcock
@kwalcock

In

override def readNative(r: Platform.Pointer): Int

I could ask r for r.equals(<pointer of py.Dynamic.global.None>), however there doesn't seem to be a way to get to the pointer of None except in the debugger.