I updated my minimal example on github, which may be useful for others trying to get off the ground.
Next, I've switched to my local install of anaconda by changing the path to python in build.sbt:
lazy val python = Python("/opt/anaconda3/bin/python3.9")and the existing example works fine.
However, when I then try to experiment with numpy at runtime a particular library can't be loaded:
[info] INTEL MKL ERROR: dlopen(/opt/anaconda3/lib/libmkl_intel_thread.1.dylib, 0x0009): Library not loaded: @rpath/libiomp5.dylibI notice that /opt/anaconda3/lib/libiomp5.dylib does exist, although /opt/anaconda3/lib/libmkl_intel_thread.1.dylib does not.
Has anyone experienced a similar problem?
I tried a few of these suggestions in the anaconda docs but unfortunately they haven't resolved my issue.
The full error message reads:
The full error message reads:
[info] INTEL MKL ERROR: dlopen(/opt/anaconda3/lib/libmkl_intel_thread.1.dylib, 0x0009): Library not loaded: @rpath/libiomp5.dylib
[info] Referenced from: /opt/anaconda3/lib/libmkl_intel_thread.1.dylib
[info] Reason: tried: '/Applications/IntelliJ IDEA CE.app/Contents/jbr/Contents/Home/bin/../lib/jli/libiomp5.dylib' (no such file), '/usr/lib/libiomp5.dylib' (no such file).
[info] Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.1.dylib.On the executable /opt/anaconda3/bin/python3.9 it would appear (from using otool) that LC_RPATH is correct:
Load command 14
cmd LC_RPATH
cmdsize 272
path /opt/anaconda3/lib (offset 12)and in /opt/anaconda3/lib/libmkl_intel_thread.1.dylib itself, I see:
Load command 10
cmd LC_LOAD_DYLIB
cmdsize 48
name @rpath/libiomp5.dylib (offset 24)I'm in a world of macos/rpath pain now and well out of my depth, but none of the above looks incorrect to me.
Would anyone care to venture why it doesn't pick up @rpath/libiomp5.dylib from /opt/anaconda3/lib?
I put the minimal anaconda/numpy example on this branch. Again, I'd be curious to know if that just works out of the box for folks with a local anaconda install on MacOS.
8 replies
Hello, I'm trying to use Scalapy to integrate Python Arima library with the Scala code from the rest of my project. It is working fine, but when I'm trying to integrate with Jenkins pipeline, I'm getting an error with Sonar analysis.
I have an class in my project that contains the Scalapy code, all others classes have Scala code.
I tried to ignore this specific file but no success, and the goal would be to analyse all Scala and ScalaPy code in SonarQube
I have an class in my project that contains the Scalapy code, all others classes have Scala code.
I tried to ignore this specific file but no success, and the goal would be to analyse all Scala and ScalaPy code in SonarQube
@alicebrb: interesting, it seems that the Sonar analysis doesn't like code that's generated by the macro. I'll see if there's some way to ensure that positions are assigned to the generated code, but in the meantime you'll probably have to disable the analysis on files that involve ScalaPy calls.
1 reply
@andyczerwonka: not yet, but things should "just work" with a container containing both SBT and a Python installation. The ScalaPy website itself is built in Netlify's Docker containers, and we run some ScalaPy code as part of that to generate the outputs of the code examples. So it works there at least.
if you end up putting together a setup for Docker, it would be great to add that to the docs!
@mn98: I noticed that you seem to be running the examples from IntelliJ. Do you run into the same error when running SBT on the command line?
it's odd, because anaconda+numpy works in isolation, scalapy works with anaconda, my branch works for at least one other user and yet anaconda+numpy won't work for me with scalapy
@mn98: hmm, that's very interesting; unfortunately I'm running on NixOS myself so testing out Anaconda locally is even trickier, but will try to investigate
@andyczerwonka: ah, in that case what you'll need to do is add an additional Docker step to install Python (3.7-3.9) and ScalaPy should automatically pick up the install
actually, this might be a bit more complicated since we typically configure the library path through SBT settings, lemme think
ok, what you'll need to do is set
javaOptions specifically for the Docker image (I believe javaOptions in Universal should work) to -Djna.library.path=$pythonLibsDir where $pythonLibsDir is replaced with the Python installation path that python3-config prints in the container
a third not yet built-in option could be to add https://github.com/kiendang/python-native-libs as a dependency and configure the library path at runtime with:
import ai.kien.python.Python
Python().scalapyProperties.fold(
ex => println(s"Error while getting ScalaPy properties: $ex"),
props => props.foreach { case(k, v) => System.setProperty(k, v) }
)
Hi @shadaj:matrix.org! Could you tell me if it is possible to use my own python libraries from GitHub through py.module(...)? I thought in doing something like py.module("git+https://github.com/my-library.git") but it didn't work.
@josea-source: you'll need to install the library using the standard
pip tool, ScalaPy just forwards to the underlying Python implementation when loading modules
@sven42: generally, readers are reserved for conversion between low-level types (think ints, bools, strings, etc.); what are you trying to read into a case class?
@shadaj:matrix.org: Thanks for your answer. I am trying to use the result of a call to a python service in the scala-js frontend. The python service returns a list of my domain class, implemented as class in python and a case class in scala. Currently I implemented my python service scala facade to return a list of domain-facades instead, which I map to the case class afterwards. This works, but I thought there must be a more direct way.
Ah, yeah that would probably need to be introduced as a utility outside of the ScalaPy core. Maybe something like https://github.com/softwaremill/magnolia could help in setting one up?
I've been experimenting with ScalaPy and it is amazing! I did try to import the python library wandb and get the following exception
[error] Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'IndexError'> list index out of range
Any ideas on why this might be happening? I'm using
- sbt 1.6.2
- python 3.7
- scala 2.12.8
wandb was installed via pip3 install wanbd and I invoked it in my application via
import me.shadaj.scalapy.py
val wandb = py.Module("wandb")thanks for any help!
1 reply
full stack trace below
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$2(CPythonInterpreter.scala:328)
[error] at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:314)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:313)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$2(CPythonInterpreter.scala:233)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$1(CPythonInterpreter.scala:230)
[error] at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error] at me.shadaj.scalapy.interpreter.CPythonInterpreter$.importModule(CPythonInterpreter.scala:229)
[error] at me.shadaj.scalapy.py.ModuleApply.apply(ModuleApply.scala:9)
[error] at me.shadaj.scalapy.py.ModuleApply.apply$(ModuleApply.scala:8)
[error] at me.shadaj.scalapy.py.Module$.apply(Module.scala:7)
[error] at me.shadaj.scalapy.py.package$.module(package.scala:14)
[error] at example.WandB$.delayedEndpoint$example$WandB$1(WandB.scala:8)
[error] at example.WandB$delayedInit$body.apply(WandB.scala:6)
[error] at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] at scala.collection.immutable.List.foreach(List.scala:392)
[error] at scala.App.main(App.scala:80)
[error] at scala.App.main$(App.scala:78)
[error] at example.WandB$.main(WandB.scala:6)
[error] at example.WandB.main(WandB.scala)
[error] Nonzero exit code returned from runner: 1
@timsetsfire: Hmm, interesting it seems that the crash is happening within
wandb when it's being imported. The Python stack traces aren't great right now when there are crashes in Python unfortunately, but let me try to see if I can reproduce locally.
FROM openjdk:11.0.12-jdk-slim-buster
RUN apt update && apt install -y python3 python3-pip python3-distutils fontconfig nginx && \
pip3 install --upgrade pip && \
pip3 install --no-cache-dir wheel setuptools && \
apt install -y python3-setuptools && \
chmod 707 /var/lib/nginx
RUN apt-get update
RUN apt-get install apt-transport-https curl gnupg sudo -yqq
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee /etc/apt/sources.list.d/sbt.list
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.list.d/sbt_old.list
RUN curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
RUN chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
RUN apt-get update
RUN apt-get install sbt -y
RUN mkdir -p /tmp/scalapy
COPY ./ /tmp/scalapy
RUN pip3 install wandb jupyter -q
Awesome, thanks for the details. My guess is that if the library works in regular Python, there may be some global state that we're not initializing properly. Going to work on setting up proper Python stack traces to help with issues like this.
@mpasa: you can do this using the
py"..." feature (https://scalapy.dev/docs/interacting-with-python#custom-python-snippets) and interpolating the function into the body
note that you won't be able to write multi-line Python code, so you may need to use the Python eval function inside to execute a larger block
Thank you! It seems PyQuote or eval can't deal with assignations. I get the following:
Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'SyntaxError'> ('invalid syntax', ('<string>', 1, 2, 'f=spy_o_0'))But I made it work like this:
import me.shadaj.scalapy.py
import py.PyQuote
def toPyValue[T](x: T)(implicit writer: Writer[T]): PyValue = writer.write(x)
def f(a: Int): Int = a*2
CPythonInterpreter.set("f", toPyValue(f _))
println(py"f(5)")
ah yeah, you won't be able to perform assignments with PyQuote since we send the Python snippet in expression mode (having an API that supports statement mode is something that should probably be implemented though)
Hi all, has anyone used/worked with pandas via ScalaPy? I tried to load a dataframe but got this error:
me.shadaj.scalapy.py.PythonException: <class 'TypeError'> 'DataFrame' object is not callable.I'm able to import pandas as a module and print its version. It fails when I tried calling pd.DataFrame.from_records. The same code in Python is working in Python REPL.
val numpy = py.module("numpy")
val data = numpy.array(Seq( (1, "a"), (2, "b") ).toPythonCopy)
val pandas = py.module("pandas")
val pdVersion = pandas.__version__
println(s"Pandas version is $pdVersion")
// The following line fails
val dataframe = pandas.DataFrame.from_records(data)
Wondering if anyone can shed some light here. I'm trying to integrate ScalaPy into our environment. We use the DockerPlugin, so I have added Python to the environment like so:
Cmd("USER", "root")
Cmd("ENV", "PYTHONUNBUFFERED 1")
Cmd("RUN", "apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python")
Cmd("RUN", "python3 -m ensurepip")
Cmd("RUN", "pip3 install --no-cache --upgrade pip setuptools")I then hit an endpoint that invokes a very simple piece of code:
def sum(acl: AccessControlSession, xs: Seq[Double]): Future[Either[SessionError, SumAnswer]] = {
import me.shadaj.scalapy.py
import me.shadaj.scalapy.py.SeqConverters
val pythonList = py.Dynamic.global.list(xs.toPythonProxy)
val sum = py.Dynamic.global.sum(pythonList).as[Double]
Future.successful(Right(SumAnswer(sum)))
}And Boom!!! I've killed the container.
| Fatal Python error: create_gil: PyCOND_INIT(gil->cond) failed
| Python runtime state: preinitialized
|
| Current thread 0x00007f43614a7640 (most recent call first):
| <no Python frame>
| #
| # A fatal error has been detected by the Java Runtime Environment:
| #
| # SIGSEGV (0xb) at pc=0x00007f445cfe793b, pid=1, tid=224
| #
| # JRE version: OpenJDK Runtime Environment Temurin-11.0.13+8 (11.0.13+8) (build 11.0.13+8)
| # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (11.0.13+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
| # Problematic frame:
| # C [libc.so.6+0x2693b] abort+0x1ed
| #
| # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /opt/docker/core.1)
| #
| # An error report file with more information is saved as:
| # /tmp/hs_err_pid1.log
| #
| # If you would like to submit a bug report, please visit:
| # https://github.com/adoptium/adoptium-support/issues
| # The crash happened outside the Java Virtual Machine in native code.
| # See problematic frame for where to report the bug.
| #Hello! I'm trying to implement a writer and reader for Option[A] and not sure exactly how. I'll want to check for NoneType in Python. What I have right now for the writer is:
implicit def optionWriter[A](implicit aWriter: Writer[A], anyWriter: Writer[py.Any]): Writer[Option[A]] = new Writer[Option[A]] {
override def write(opt: Option[A]) = {
opt match {
case Some(a) => aWriter.write(a)
case None => anyWriter.write(py.eval("None"))
}
}
}The py.eval part is a bit shady, and not sure if there is a better way. For the reader, not sure how to start checking if a Platform.Pointer or PyValue is None.
Hi all, I'm pointing SCALAPY_PYTHON_PROGRAMNAME to a virtualenv's python3 path (e.g. /Users/foo/.virtualenvs/scalapy/bin/python3). It's working if I do py.module("numpy") as I have numpy installed in the virtualenv. The directory of the actual virtualenv instance is at /Users/foo/python/venvs/scalapy.
Now, I want to load a Python script (foo.py) under a sub-directory test in that virtualenv directory. (The full path is /Users/foo/python/venvs/scalapy/test/foo.py) but I can't figure out how to load test.foo successfully. The py.module("test.foo") call is failing. How do I call py.module to load test.foo module? Thanks in advance!
@mpasa: the Reader/Writer API is actually ScalaPy internal, and isn't intended to be implemented by end-users. If you want richer, higher-level conversions, you should probably define your own Reader/Writer traits that convert between Scala values and the higher-level py.Any subtypes.
We should probably document this / only allow ScalaPy internal code to implement these traits.
@voonchav_gitlab: have you set up an
__init__.py file in the test folder? you'll need that so that Python recognizes it as a module