Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 29 17:57
    shadaj milestoned #312
  • Sep 29 17:57
    shadaj milestoned #312
  • Sep 29 17:57

    shadaj on main

    chore: add a clear error while … (compare)

  • Sep 29 17:57
    shadaj closed #312
  • Sep 29 08:03
    cric96 synchronize #312
  • Sep 29 04:27
    kiendang commented #303
  • Sep 28 18:54

    shadaj on fix-empty-parens-scala-3-facade

    (compare)

  • Sep 28 18:54
    shadaj closed #311
  • Sep 28 18:54

    shadaj on main

    Ensure that facade methods with… (compare)

  • Sep 28 18:54
    shadaj closed #313
  • Sep 28 18:43
    shadaj synchronize #313
  • Sep 28 18:43

    shadaj on fix-empty-parens-scala-3-facade

    Update changelog (compare)

  • Sep 28 18:42
    shadaj milestoned #313
  • Sep 28 18:42
    shadaj milestoned #313
  • Sep 28 18:42
    shadaj opened #313
  • Sep 28 18:41

    shadaj on fix-empty-parens-scala-3-facade

    Ensure that facade methods with… (compare)

  • Sep 28 17:56
    shadaj closed #282
  • Sep 28 17:56
    shadaj commented #282
  • Sep 28 17:55
    shadaj commented #303
  • Sep 28 13:02
    cric96 opened #312
mn98
@mn98
it's odd, because anaconda+numpy works in isolation, scalapy works with anaconda, my branch works for at least one other user and yet anaconda+numpy won't work for me with scalapy
Andy Czerwonka
@andyczerwonka
@shadaj:matrix.org Today, we don't ship SBT as part of our container, we package things up using via .enablePlugins(DockerPlugin, JavaAgent, JavaAppPackaging, AshScriptPlugin).
shadaj
@shadaj:matrix.org
[m]
@mn98: hmm, that's very interesting; unfortunately I'm running on NixOS myself so testing out Anaconda locally is even trickier, but will try to investigate
@andyczerwonka: ah, in that case what you'll need to do is add an additional Docker step to install Python (3.7-3.9) and ScalaPy should automatically pick up the install
Andy Czerwonka
@andyczerwonka
:thumbsup:
shadaj
@shadaj:matrix.org
[m]
actually, this might be a bit more complicated since we typically configure the library path through SBT settings, lemme think
ok, what you'll need to do is set javaOptions specifically for the Docker image (I believe javaOptions in Universal should work) to -Djna.library.path=$pythonLibsDir where $pythonLibsDir is replaced with the Python installation path that python3-config prints in the container
a third not yet built-in option could be to add https://github.com/kiendang/python-native-libs as a dependency and configure the library path at runtime with:
import ai.kien.python.Python

Python().scalapyProperties.fold(
  ex => println(s"Error while getting ScalaPy properties: $ex"),
  props => props.foreach { case(k, v) => System.setProperty(k, v) }
)
Andy Czerwonka
@andyczerwonka
I'll give it a go, and if/when we get it working, I'd be happy to issue a PR for docs
josea-source
@josea-source
Hi @shadaj:matrix.org! Could you tell me if it is possible to use my own python libraries from GitHub through py.module(...)? I thought in doing something like py.module("git+https://github.com/my-library.git") but it didn't work.
sven42
@sven42
Hi, I am trying to get started with scalapy. Is there any documentation or an example on how to implement a Reader for a case class? Is the case class allowed to nest another case case class (with its own Reader of course). Thanks for any help.
shadaj
@shadaj:matrix.org
[m]
@josea-source: you'll need to install the library using the standard pip tool, ScalaPy just forwards to the underlying Python implementation when loading modules
@sven42: generally, readers are reserved for conversion between low-level types (think ints, bools, strings, etc.); what are you trying to read into a case class?
sven42
@sven42
@shadaj:matrix.org: Thanks for your answer. I am trying to use the result of a call to a python service in the scala-js frontend. The python service returns a list of my domain class, implemented as class in python and a case class in scala. Currently I implemented my python service scala facade to return a list of domain-facades instead, which I map to the case class afterwards. This works, but I thought there must be a more direct way.
shadaj
@shadaj:matrix.org
[m]
Ah, yeah that would probably need to be introduced as a utility outside of the ScalaPy core. Maybe something like https://github.com/softwaremill/magnolia could help in setting one up?
Timothy Whittaker
@timsetsfire

I've been experimenting with ScalaPy and it is amazing! I did try to import the python library wandb and get the following exception

[error] Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'IndexError'> list index out of range

Any ideas on why this might be happening? I'm using

  • sbt 1.6.2
  • python 3.7
  • scala 2.12.8

wandb was installed via pip3 install wanbd and I invoked it in my application via

import me.shadaj.scalapy.py
val wandb = py.Module("wandb")

thanks for any help!

1 reply
full stack trace below
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$2(CPythonInterpreter.scala:328)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:314)
[error]     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:313)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$2(CPythonInterpreter.scala:233)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$1(CPythonInterpreter.scala:230)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.importModule(CPythonInterpreter.scala:229)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply(ModuleApply.scala:9)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply$(ModuleApply.scala:8)
[error]     at me.shadaj.scalapy.py.Module$.apply(Module.scala:7)
[error]     at me.shadaj.scalapy.py.package$.module(package.scala:14)
[error]     at example.WandB$.delayedEndpoint$example$WandB$1(WandB.scala:8)
[error]     at example.WandB$delayedInit$body.apply(WandB.scala:6)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.WandB$.main(WandB.scala:6)
[error]     at example.WandB.main(WandB.scala)
[error] Nonzero exit code returned from runner: 1
shadaj
@shadaj:matrix.org
[m]
@timsetsfire: Hmm, interesting it seems that the crash is happening within wandb when it's being imported. The Python stack traces aren't great right now when there are crashes in Python unfortunately, but let me try to see if I can reproduce locally.
Timothy Whittaker
@timsetsfire
here is my dockerfile
FROM openjdk:11.0.12-jdk-slim-buster
RUN apt update && apt install -y python3 python3-pip python3-distutils fontconfig nginx && \
    pip3 install --upgrade pip && \
    pip3 install --no-cache-dir wheel setuptools && \
    apt install -y python3-setuptools && \
    chmod 707 /var/lib/nginx
RUN apt-get update
RUN apt-get install apt-transport-https curl gnupg sudo -yqq
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee /etc/apt/sources.list.d/sbt.list
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.list.d/sbt_old.list
RUN curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
RUN chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
RUN apt-get update
RUN apt-get install sbt -y

RUN mkdir -p /tmp/scalapy 
COPY ./ /tmp/scalapy
RUN pip3 install wandb jupyter -q
the COPY is just pulling in my scala project and then i was interested with the container via bash.
shadaj
@shadaj:matrix.org
[m]
Awesome, thanks for the details. My guess is that if the library works in regular Python, there may be some global state that we're not initializing properly. Going to work on setting up proper Python stack traces to help with issues like this.
Miguel Pérez Pasalodos
@mpasa

Hello everyone! Do you know if I can achieve something like this in Scalapy?

  1. Define a function in Scala
  2. Eval an arbitrary Python string that can use that function.

For example:

  def f(a: Int): Int = a*2

  py.eval("""
  a = 5
  f(5)
  """)
shadaj
@shadaj:matrix.org
[m]
@mpasa: you can do this using the py"..." feature (https://scalapy.dev/docs/interacting-with-python#custom-python-snippets) and interpolating the function into the body
note that you won't be able to write multi-line Python code, so you may need to use the Python eval function inside to execute a larger block
Miguel Pérez Pasalodos
@mpasa

Thank you! It seems PyQuote or eval can't deal with assignations. I get the following:

 Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SyntaxError'> ('invalid syntax', ('<string>', 1, 2, 'f=spy_o_0'))

But I made it work like this:

import me.shadaj.scalapy.py
import py.PyQuote

def toPyValue[T](x: T)(implicit writer: Writer[T]): PyValue = writer.write(x)
def f(a: Int): Int = a*2
CPythonInterpreter.set("f",  toPyValue(f _))
println(py"f(5)")
shadaj
@shadaj:matrix.org
[m]
ah yeah, you won't be able to perform assignments with PyQuote since we send the Python snippet in expression mode (having an API that supports statement mode is something that should probably be implemented though)
vonchav
@voonchav_gitlab

Hi all, has anyone used/worked with pandas via ScalaPy? I tried to load a dataframe but got this error:

me.shadaj.scalapy.py.PythonException: <class 'TypeError'> 'DataFrame' object is not callable.

I'm able to import pandas as a module and print its version. It fails when I tried calling pd.DataFrame.from_records. The same code in Python is working in Python REPL.

val numpy = py.module("numpy")
val data = numpy.array(Seq( (1, "a"), (2, "b") ).toPythonCopy)

val pandas = py.module("pandas")

val pdVersion = pandas.__version__
println(s"Pandas version is $pdVersion")

// The following line fails
val dataframe = pandas.DataFrame.from_records(data)
vonchav
@voonchav_gitlab
^ Actually this info is not incomplete/incorrect. Sorry. I found what is causing this. It wasn't ScalaPy. :)
Andy Czerwonka
@andyczerwonka

Wondering if anyone can shed some light here. I'm trying to integrate ScalaPy into our environment. We use the DockerPlugin, so I have added Python to the environment like so:

Cmd("USER", "root")
Cmd("ENV", "PYTHONUNBUFFERED 1")
Cmd("RUN", "apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python")
Cmd("RUN", "python3 -m ensurepip")
Cmd("RUN", "pip3 install --no-cache --upgrade pip setuptools")

I then hit an endpoint that invokes a very simple piece of code:

  def sum(acl: AccessControlSession, xs: Seq[Double]): Future[Either[SessionError, SumAnswer]] = {
    import me.shadaj.scalapy.py
    import me.shadaj.scalapy.py.SeqConverters
    val pythonList = py.Dynamic.global.list(xs.toPythonProxy)
    val sum = py.Dynamic.global.sum(pythonList).as[Double]
    Future.successful(Right(SumAnswer(sum)))
  }

And Boom!!! I've killed the container.

| Fatal Python error: create_gil: PyCOND_INIT(gil->cond) failed
| Python runtime state: preinitialized
|
| Current thread 0x00007f43614a7640 (most recent call first):
| <no Python frame>
| #
| # A fatal error has been detected by the Java Runtime Environment:
| #
| #  SIGSEGV (0xb) at pc=0x00007f445cfe793b, pid=1, tid=224
| #
| # JRE version: OpenJDK Runtime Environment Temurin-11.0.13+8 (11.0.13+8) (build 11.0.13+8)
| # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (11.0.13+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
| # Problematic frame:
| # C  [libc.so.6+0x2693b]  abort+0x1ed
| #
| # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /opt/docker/core.1)
| #
| # An error report file with more information is saved as:
| # /tmp/hs_err_pid1.log
| #
| # If you would like to submit a bug report, please visit:
| #   https://github.com/adoptium/adoptium-support/issues
| # The crash happened outside the Java Virtual Machine in native code.
| # See problematic frame for where to report the bug.
| #
1 reply
Miguel Pérez Pasalodos
@mpasa

Hello! I'm trying to implement a writer and reader for Option[A] and not sure exactly how. I'll want to check for NoneType in Python. What I have right now for the writer is:

implicit def optionWriter[A](implicit aWriter: Writer[A], anyWriter: Writer[py.Any]): Writer[Option[A]] = new Writer[Option[A]] {
  override def write(opt: Option[A]) = {
    opt match {
      case Some(a) => aWriter.write(a)
      case None => anyWriter.write(py.eval("None"))
    }
  }
}

The py.eval part is a bit shady, and not sure if there is a better way. For the reader, not sure how to start checking if a Platform.Pointer or PyValue is None.

vonchav
@voonchav_gitlab

Hi all, I'm pointing SCALAPY_PYTHON_PROGRAMNAME to a virtualenv's python3 path (e.g. /Users/foo/.virtualenvs/scalapy/bin/python3). It's working if I do py.module("numpy") as I have numpy installed in the virtualenv. The directory of the actual virtualenv instance is at /Users/foo/python/venvs/scalapy.

Now, I want to load a Python script (foo.py) under a sub-directory test in that virtualenv directory. (The full path is /Users/foo/python/venvs/scalapy/test/foo.py) but I can't figure out how to load test.foo successfully. The py.module("test.foo") call is failing. How do I call py.module to load test.foo module? Thanks in advance!

shadaj
@shadaj:matrix.org
[m]
@mpasa: the Reader/Writer API is actually ScalaPy internal, and isn't intended to be implemented by end-users. If you want richer, higher-level conversions, you should probably define your own Reader/Writer traits that convert between Scala values and the higher-level py.Any subtypes.
We should probably document this / only allow ScalaPy internal code to implement these traits.
@voonchav_gitlab: have you set up an __init__.py file in the test folder? you'll need that so that Python recognizes it as a module
Miguel Pérez Pasalodos
@mpasa
@shadaj:matrix.org thanks! I have another question. Is it possible to run different interpreters in the same JVM? It doesn't look possible as CPythonInterpreter is an object
shadaj
@shadaj:matrix.org
[m]
@mpasa: unfortunately not, the story with subinterpreters with the native CPython APIs is quite a mess (https://docs.python.org/3/c-api/init.html#bugs-and-caveats) so in ScalaPy we chose to focus on just a single interpreter instance
however, ScalaPy does support safe multithreaded access to Python values
and if you're using a Python library backed by native code, you might be able to get good scaling if that native code appropriately releases the GIL
Gabriel Claramunt
@gclaramunt
What type I need to use in Scala to get a Python's dict ?
Gabriel Claramunt
@gclaramunt
(for now, I'm using a list and convert it to dict on the python side, but I'm not totally happy changing the python side)
shadaj
@shadaj:matrix.org
[m]
@gclaramunt: you can read it as a mutable.Map[K, V]!
Gabriel Claramunt
@gclaramunt
nice... and how I pass a Map to python? mutable.Map too?
shadaj
@shadaj:matrix.org
[m]
for sending a map to Python you'll need an immutable map (because we don't currently support proxies from Python back into Scala for maps)
Gabriel Claramunt
@gclaramunt
how do I do the conversion? toPythonProxy ?
Gabriel Claramunt
@gclaramunt
(the library is awesome, btw)
shadaj
@shadaj:matrix.org
[m]
it will automatically happen! (since there's no choice between proxy and copy)
that's great to hear! please let us know if there are any things we can do to make it more awesome!
Gabriel Claramunt
@gclaramunt
Can I pass an arbitrary object created in Scala as a parameter to a python call? Couldn't find any example
(I might get away with just passing a Map, but anyway)
shadaj
@shadaj:matrix.org
[m]

Not arbitrary objects at the moment. There have been discussions about making this possible through some type of annotation eventually, but it will be some time before it's available.

In the meantime, yeah passing maps is probably the way to go.