Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 02 17:46
    scala-steward closed #281
  • Aug 02 17:46
    scala-steward commented #281
  • Aug 02 17:46
    scala-steward opened #293
  • Jul 29 20:13
    scala-steward closed #290
  • Jul 29 20:13
    scala-steward commented #290
  • Jul 29 20:12
    scala-steward opened #292
  • Jul 27 17:53
    scala-steward opened #291
  • Jul 10 03:59
    scala-steward closed #272
  • Jul 10 03:59
    scala-steward commented #272
  • Jul 10 03:59
    scala-steward opened #290
  • Jul 08 05:53
    scala-steward closed #268
  • Jul 08 05:53
    scala-steward commented #268
  • Jul 08 05:53
    scala-steward opened #289
  • Jul 08 05:53
    scala-steward closed #277
  • Jul 08 05:53
    scala-steward commented #277
  • Jul 08 05:53
    scala-steward opened #288
  • Jul 08 05:53
    scala-steward opened #287
  • Jul 08 05:53
    scala-steward opened #286
  • Jun 23 19:13
    sfriedowitz opened #285
  • May 24 20:17
    netlify[bot] commented #284
mn98
@mn98

On the executable /opt/anaconda3/bin/python3.9 it would appear (from using otool) that LC_RPATH is correct:

Load command 14
          cmd LC_RPATH
      cmdsize 272
         path /opt/anaconda3/lib (offset 12)

and in /opt/anaconda3/lib/libmkl_intel_thread.1.dylib itself, I see:

Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libiomp5.dylib (offset 24)

I'm in a world of macos/rpath pain now and well out of my depth, but none of the above looks incorrect to me.
Would anyone care to venture why it doesn't pick up @rpath/libiomp5.dylib from /opt/anaconda3/lib?

mn98
@mn98
I put the minimal anaconda/numpy example on this branch. Again, I'd be curious to know if that just works out of the box for folks with a local anaconda install on MacOS.
8 replies
alicebrb
@alicebrb
Hello, I'm trying to use Scalapy to integrate Python Arima library with the Scala code from the rest of my project. It is working fine, but when I'm trying to integrate with Jenkins pipeline, I'm getting an error with Sonar analysis.
I have an class in my project that contains the Scalapy code, all others classes have Scala code.
I tried to ignore this specific file but no success, and the goal would be to analyse all Scala and ScalaPy code in SonarQube
image.png
Andy Czerwonka
@andyczerwonka
Looking at the Getting Started docs, I'm curious if there are some details somewhere around how to get this up and running in the docker container?
shadaj
@shadaj:matrix.org
[m]
@alicebrb: interesting, it seems that the Sonar analysis doesn't like code that's generated by the macro. I'll see if there's some way to ensure that positions are assigned to the generated code, but in the meantime you'll probably have to disable the analysis on files that involve ScalaPy calls.
1 reply
@andyczerwonka: not yet, but things should "just work" with a container containing both SBT and a Python installation. The ScalaPy website itself is built in Netlify's Docker containers, and we run some ScalaPy code as part of that to generate the outputs of the code examples. So it works there at least.
if you end up putting together a setup for Docker, it would be great to add that to the docs!
@mn98: I noticed that you seem to be running the examples from IntelliJ. Do you run into the same error when running SBT on the command line?
mn98
@mn98
@shadaj:matrix.org yes, unfortunately the error is identical via either sbt on the command line or the sbt console within IntelliJ
it's odd, because anaconda+numpy works in isolation, scalapy works with anaconda, my branch works for at least one other user and yet anaconda+numpy won't work for me with scalapy
Andy Czerwonka
@andyczerwonka
@shadaj:matrix.org Today, we don't ship SBT as part of our container, we package things up using via .enablePlugins(DockerPlugin, JavaAgent, JavaAppPackaging, AshScriptPlugin).
shadaj
@shadaj:matrix.org
[m]
@mn98: hmm, that's very interesting; unfortunately I'm running on NixOS myself so testing out Anaconda locally is even trickier, but will try to investigate
@andyczerwonka: ah, in that case what you'll need to do is add an additional Docker step to install Python (3.7-3.9) and ScalaPy should automatically pick up the install
Andy Czerwonka
@andyczerwonka
:thumbsup:
shadaj
@shadaj:matrix.org
[m]
actually, this might be a bit more complicated since we typically configure the library path through SBT settings, lemme think
ok, what you'll need to do is set javaOptions specifically for the Docker image (I believe javaOptions in Universal should work) to -Djna.library.path=$pythonLibsDir where $pythonLibsDir is replaced with the Python installation path that python3-config prints in the container
a third not yet built-in option could be to add https://github.com/kiendang/python-native-libs as a dependency and configure the library path at runtime with:
import ai.kien.python.Python

Python().scalapyProperties.fold(
  ex => println(s"Error while getting ScalaPy properties: $ex"),
  props => props.foreach { case(k, v) => System.setProperty(k, v) }
)
Andy Czerwonka
@andyczerwonka
I'll give it a go, and if/when we get it working, I'd be happy to issue a PR for docs
josea-source
@josea-source
Hi @shadaj:matrix.org! Could you tell me if it is possible to use my own python libraries from GitHub through py.module(...)? I thought in doing something like py.module("git+https://github.com/my-library.git") but it didn't work.
sven42
@sven42
Hi, I am trying to get started with scalapy. Is there any documentation or an example on how to implement a Reader for a case class? Is the case class allowed to nest another case case class (with its own Reader of course). Thanks for any help.
shadaj
@shadaj:matrix.org
[m]
@josea-source: you'll need to install the library using the standard pip tool, ScalaPy just forwards to the underlying Python implementation when loading modules
@sven42: generally, readers are reserved for conversion between low-level types (think ints, bools, strings, etc.); what are you trying to read into a case class?
sven42
@sven42
@shadaj:matrix.org: Thanks for your answer. I am trying to use the result of a call to a python service in the scala-js frontend. The python service returns a list of my domain class, implemented as class in python and a case class in scala. Currently I implemented my python service scala facade to return a list of domain-facades instead, which I map to the case class afterwards. This works, but I thought there must be a more direct way.
shadaj
@shadaj:matrix.org
[m]
Ah, yeah that would probably need to be introduced as a utility outside of the ScalaPy core. Maybe something like https://github.com/softwaremill/magnolia could help in setting one up?
Timothy Whittaker
@timsetsfire

I've been experimenting with ScalaPy and it is amazing! I did try to import the python library wandb and get the following exception

[error] Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'IndexError'> list index out of range

Any ideas on why this might be happening? I'm using

  • sbt 1.6.2
  • python 3.7
  • scala 2.12.8

wandb was installed via pip3 install wanbd and I invoked it in my application via

import me.shadaj.scalapy.py
val wandb = py.Module("wandb")

thanks for any help!

1 reply
full stack trace below
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$2(CPythonInterpreter.scala:328)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:314)
[error]     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:313)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$2(CPythonInterpreter.scala:233)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$1(CPythonInterpreter.scala:230)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.importModule(CPythonInterpreter.scala:229)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply(ModuleApply.scala:9)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply$(ModuleApply.scala:8)
[error]     at me.shadaj.scalapy.py.Module$.apply(Module.scala:7)
[error]     at me.shadaj.scalapy.py.package$.module(package.scala:14)
[error]     at example.WandB$.delayedEndpoint$example$WandB$1(WandB.scala:8)
[error]     at example.WandB$delayedInit$body.apply(WandB.scala:6)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.WandB$.main(WandB.scala:6)
[error]     at example.WandB.main(WandB.scala)
[error] Nonzero exit code returned from runner: 1
shadaj
@shadaj:matrix.org
[m]
@timsetsfire: Hmm, interesting it seems that the crash is happening within wandb when it's being imported. The Python stack traces aren't great right now when there are crashes in Python unfortunately, but let me try to see if I can reproduce locally.
Timothy Whittaker
@timsetsfire
here is my dockerfile
FROM openjdk:11.0.12-jdk-slim-buster
RUN apt update && apt install -y python3 python3-pip python3-distutils fontconfig nginx && \
    pip3 install --upgrade pip && \
    pip3 install --no-cache-dir wheel setuptools && \
    apt install -y python3-setuptools && \
    chmod 707 /var/lib/nginx
RUN apt-get update
RUN apt-get install apt-transport-https curl gnupg sudo -yqq
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee /etc/apt/sources.list.d/sbt.list
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.list.d/sbt_old.list
RUN curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
RUN chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
RUN apt-get update
RUN apt-get install sbt -y

RUN mkdir -p /tmp/scalapy 
COPY ./ /tmp/scalapy
RUN pip3 install wandb jupyter -q
the COPY is just pulling in my scala project and then i was interested with the container via bash.
shadaj
@shadaj:matrix.org
[m]
Awesome, thanks for the details. My guess is that if the library works in regular Python, there may be some global state that we're not initializing properly. Going to work on setting up proper Python stack traces to help with issues like this.
Miguel Pérez Pasalodos
@mpasa

Hello everyone! Do you know if I can achieve something like this in Scalapy?

  1. Define a function in Scala
  2. Eval an arbitrary Python string that can use that function.

For example:

  def f(a: Int): Int = a*2

  py.eval("""
  a = 5
  f(5)
  """)
shadaj
@shadaj:matrix.org
[m]
@mpasa: you can do this using the py"..." feature (https://scalapy.dev/docs/interacting-with-python#custom-python-snippets) and interpolating the function into the body
note that you won't be able to write multi-line Python code, so you may need to use the Python eval function inside to execute a larger block
Miguel Pérez Pasalodos
@mpasa

Thank you! It seems PyQuote or eval can't deal with assignations. I get the following:

 Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SyntaxError'> ('invalid syntax', ('<string>', 1, 2, 'f=spy_o_0'))

But I made it work like this:

import me.shadaj.scalapy.py
import py.PyQuote

def toPyValue[T](x: T)(implicit writer: Writer[T]): PyValue = writer.write(x)
def f(a: Int): Int = a*2
CPythonInterpreter.set("f",  toPyValue(f _))
println(py"f(5)")
shadaj
@shadaj:matrix.org
[m]
ah yeah, you won't be able to perform assignments with PyQuote since we send the Python snippet in expression mode (having an API that supports statement mode is something that should probably be implemented though)
vonchav
@voonchav_gitlab

Hi all, has anyone used/worked with pandas via ScalaPy? I tried to load a dataframe but got this error:

me.shadaj.scalapy.py.PythonException: <class 'TypeError'> 'DataFrame' object is not callable.

I'm able to import pandas as a module and print its version. It fails when I tried calling pd.DataFrame.from_records. The same code in Python is working in Python REPL.

val numpy = py.module("numpy")
val data = numpy.array(Seq( (1, "a"), (2, "b") ).toPythonCopy)

val pandas = py.module("pandas")

val pdVersion = pandas.__version__
println(s"Pandas version is $pdVersion")

// The following line fails
val dataframe = pandas.DataFrame.from_records(data)
vonchav
@voonchav_gitlab
^ Actually this info is not incomplete/incorrect. Sorry. I found what is causing this. It wasn't ScalaPy. :)
Andy Czerwonka
@andyczerwonka

Wondering if anyone can shed some light here. I'm trying to integrate ScalaPy into our environment. We use the DockerPlugin, so I have added Python to the environment like so:

Cmd("USER", "root")
Cmd("ENV", "PYTHONUNBUFFERED 1")
Cmd("RUN", "apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python")
Cmd("RUN", "python3 -m ensurepip")
Cmd("RUN", "pip3 install --no-cache --upgrade pip setuptools")

I then hit an endpoint that invokes a very simple piece of code:

  def sum(acl: AccessControlSession, xs: Seq[Double]): Future[Either[SessionError, SumAnswer]] = {
    import me.shadaj.scalapy.py
    import me.shadaj.scalapy.py.SeqConverters
    val pythonList = py.Dynamic.global.list(xs.toPythonProxy)
    val sum = py.Dynamic.global.sum(pythonList).as[Double]
    Future.successful(Right(SumAnswer(sum)))
  }

And Boom!!! I've killed the container.

| Fatal Python error: create_gil: PyCOND_INIT(gil->cond) failed
| Python runtime state: preinitialized
|
| Current thread 0x00007f43614a7640 (most recent call first):
| <no Python frame>
| #
| # A fatal error has been detected by the Java Runtime Environment:
| #
| #  SIGSEGV (0xb) at pc=0x00007f445cfe793b, pid=1, tid=224
| #
| # JRE version: OpenJDK Runtime Environment Temurin-11.0.13+8 (11.0.13+8) (build 11.0.13+8)
| # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (11.0.13+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
| # Problematic frame:
| # C  [libc.so.6+0x2693b]  abort+0x1ed
| #
| # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /opt/docker/core.1)
| #
| # An error report file with more information is saved as:
| # /tmp/hs_err_pid1.log
| #
| # If you would like to submit a bug report, please visit:
| #   https://github.com/adoptium/adoptium-support/issues
| # The crash happened outside the Java Virtual Machine in native code.
| # See problematic frame for where to report the bug.
| #
1 reply
Miguel Pérez Pasalodos
@mpasa

Hello! I'm trying to implement a writer and reader for Option[A] and not sure exactly how. I'll want to check for NoneType in Python. What I have right now for the writer is:

implicit def optionWriter[A](implicit aWriter: Writer[A], anyWriter: Writer[py.Any]): Writer[Option[A]] = new Writer[Option[A]] {
  override def write(opt: Option[A]) = {
    opt match {
      case Some(a) => aWriter.write(a)
      case None => anyWriter.write(py.eval("None"))
    }
  }
}

The py.eval part is a bit shady, and not sure if there is a better way. For the reader, not sure how to start checking if a Platform.Pointer or PyValue is None.

vonchav
@voonchav_gitlab

Hi all, I'm pointing SCALAPY_PYTHON_PROGRAMNAME to a virtualenv's python3 path (e.g. /Users/foo/.virtualenvs/scalapy/bin/python3). It's working if I do py.module("numpy") as I have numpy installed in the virtualenv. The directory of the actual virtualenv instance is at /Users/foo/python/venvs/scalapy.

Now, I want to load a Python script (foo.py) under a sub-directory test in that virtualenv directory. (The full path is /Users/foo/python/venvs/scalapy/test/foo.py) but I can't figure out how to load test.foo successfully. The py.module("test.foo") call is failing. How do I call py.module to load test.foo module? Thanks in advance!

shadaj
@shadaj:matrix.org
[m]
@mpasa: the Reader/Writer API is actually ScalaPy internal, and isn't intended to be implemented by end-users. If you want richer, higher-level conversions, you should probably define your own Reader/Writer traits that convert between Scala values and the higher-level py.Any subtypes.
We should probably document this / only allow ScalaPy internal code to implement these traits.
@voonchav_gitlab: have you set up an __init__.py file in the test folder? you'll need that so that Python recognizes it as a module
Miguel Pérez Pasalodos
@mpasa
@shadaj:matrix.org thanks! I have another question. Is it possible to run different interpreters in the same JVM? It doesn't look possible as CPythonInterpreter is an object
shadaj
@shadaj:matrix.org
[m]
@mpasa: unfortunately not, the story with subinterpreters with the native CPython APIs is quite a mess (https://docs.python.org/3/c-api/init.html#bugs-and-caveats) so in ScalaPy we chose to focus on just a single interpreter instance
however, ScalaPy does support safe multithreaded access to Python values
and if you're using a Python library backed by native code, you might be able to get good scaling if that native code appropriately releases the GIL
Gabriel Claramunt
@gclaramunt
What type I need to use in Scala to get a Python's dict ?
Gabriel Claramunt
@gclaramunt
(for now, I'm using a list and convert it to dict on the python side, but I'm not totally happy changing the python side)