Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 23 19:13
    sfriedowitz opened #285
  • May 24 20:17
    netlify[bot] commented #284
  • May 24 20:12
    netlify[bot] commented #284
  • May 24 20:12
    scala-steward opened #284
  • May 13 07:52
    kwalcock commented #251
  • May 12 17:49
    i10416 closed #240
  • May 09 16:25
    sfriedowitz closed #283
  • May 09 16:25
    sfriedowitz commented #283
  • May 09 16:25
    sfriedowitz commented #283
  • May 06 21:04
    sfriedowitz commented #283
  • May 06 20:43
    shadaj commented #283
  • May 06 15:28
    kwalcock commented #251
  • May 05 23:17
    sfriedowitz commented #283
  • May 05 23:14
    sfriedowitz commented #283
  • May 05 21:34
    shadaj commented #283
  • May 04 23:25
    sfriedowitz edited #283
  • May 04 23:24
    sfriedowitz edited #283
  • May 04 23:21
    sfriedowitz opened #283
  • May 01 00:13
    kiendang commented #280
  • Apr 30 23:59
    kiendang commented #280
Darsh Selarka
@darshselarka1497
@shadaj:matrix.org This is the output for the class type
class me.shadaj.scalapy.py.AnyDynamics$$anon$15$$anon$16
shadaj
@shadaj:matrix.org
[m]
@darshselarka1497: hmm, that's weird, in theory things should compile then; maybe you can try storing py_spark_sql.SparkSession in a variable before using it?
Pascal Méheut
@pascal_meheut_gitlab
Hi. I'm running Scalapy with Python 3.9 in Anaconda on a Mac. It works fine. I just had to configure the jna.library.path manually because python3-config returns something wrong. But some modules cannot be imported. numpy and pandas work fine but when I try to import feather or xgboost, I got a message
"Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'ModuleNotFoundError'> No module named 'xgboost'"
shadaj
@shadaj:matrix.org
[m]
@pascal_meheut_gitlab: hmm, these packages are installed in your Anaconda environment as well? those should work out of the box; if you're using a virtualenv you need to follow the instructions at https://scalapy.dev/docs/#virtualenv though
Pascal Méheut
@pascal_meheut_gitlab
Yes, this package are installed: they are my bread and butter. I'm not using VirtualEnv at all, just Anaconda.
shadaj
@shadaj:matrix.org
[m]
that's surprising, is the installation of xgboost right next to numpy and the modules that do work?
Pascal Méheut
@pascal_meheut_gitlab
Yes. Everything is in $HOME/opt/anaconda3/envs/lbo/lib/python3.9/site-packages
lbo being my environment name. I'll test on another Mac and on Linux & Windows tomorrow.
Pascal Méheut
@pascal_meheut_gitlab
Ok, this was a problem with my installation. I removed Anaconda, reinstalled it, recreated the environment and now it works. Thanks.
Pascal Méheut
@pascal_meheut_gitlab
Another question: anybody wrote a facade or explained how to use a Pandas dataframe?
mn98
@mn98
Hi all, I'm trying to get started with ScalaPy and experiencing issues similar to others with respect to libraries not being found.
I've tried a few of the solutions proposed but have been miserably unsuccessful in getting it to work. What's slightly different about my setup is that I've used pyenv to install python 3.9.9 and anaconda 3-2011.11. Has anyone had experience with this approach and be able to share any pointers? Many thanks in advance!
mn98
@mn98

To make this slightly easier, I've removed pyenv from the equation and pushed this skeleton example to github.
At this point, upon executing runMain hello in the sbt shell the error begins with:

java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
dlopen(libpython3.dylib, 0x0009): tried: '/Applications/IntelliJ IDEA CE.app/Contents/jbr/Contents/Home/bin/../lib/jli/libpython3.dylib' (no such file) ...

And it's correct, that file doesn't exist, it's actually /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9.dylib, but how do I get this to be picked up?

1 reply
I get the same link error when running either in the sbt shell or within IntelliJ IDEA, so I don't think it's an IDE issue.
mn98
@mn98
I'd be curious to know if what I've pushed to github works out of the box for people.
Kien Dang
@kiendang
add fork := true to your build.sbt and things should work fine
1 reply
mn98
@mn98
@kiendang thank you very much!!
mn98
@mn98
I updated my minimal example on github, which may be useful for others trying to get off the ground.
mn98
@mn98

Next, I've switched to my local install of anaconda by changing the path to python in build.sbt:

lazy val python = Python("/opt/anaconda3/bin/python3.9")

and the existing example works fine.
However, when I then try to experiment with numpy at runtime a particular library can't be loaded:

[info] INTEL MKL ERROR: dlopen(/opt/anaconda3/lib/libmkl_intel_thread.1.dylib, 0x0009): Library not loaded: @rpath/libiomp5.dylib

I notice that /opt/anaconda3/lib/libiomp5.dylib does exist, although /opt/anaconda3/lib/libmkl_intel_thread.1.dylib does not.
Has anyone experienced a similar problem?

mn98
@mn98
Correction, both libraries are present under /opt/anaconda3/lib yet they are not loaded at runtime.
mn98
@mn98
I tried a few of these suggestions in the anaconda docs but unfortunately they haven't resolved my issue.
The full error message reads:
[info] INTEL MKL ERROR: dlopen(/opt/anaconda3/lib/libmkl_intel_thread.1.dylib, 0x0009): Library not loaded: @rpath/libiomp5.dylib
[info]   Referenced from: /opt/anaconda3/lib/libmkl_intel_thread.1.dylib
[info]   Reason: tried: '/Applications/IntelliJ IDEA CE.app/Contents/jbr/Contents/Home/bin/../lib/jli/libiomp5.dylib' (no such file), '/usr/lib/libiomp5.dylib' (no such file).
[info] Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.1.dylib.
mn98
@mn98

On the executable /opt/anaconda3/bin/python3.9 it would appear (from using otool) that LC_RPATH is correct:

Load command 14
          cmd LC_RPATH
      cmdsize 272
         path /opt/anaconda3/lib (offset 12)

and in /opt/anaconda3/lib/libmkl_intel_thread.1.dylib itself, I see:

Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libiomp5.dylib (offset 24)

I'm in a world of macos/rpath pain now and well out of my depth, but none of the above looks incorrect to me.
Would anyone care to venture why it doesn't pick up @rpath/libiomp5.dylib from /opt/anaconda3/lib?

mn98
@mn98
I put the minimal anaconda/numpy example on this branch. Again, I'd be curious to know if that just works out of the box for folks with a local anaconda install on MacOS.
8 replies
alicebrb
@alicebrb
Hello, I'm trying to use Scalapy to integrate Python Arima library with the Scala code from the rest of my project. It is working fine, but when I'm trying to integrate with Jenkins pipeline, I'm getting an error with Sonar analysis.
I have an class in my project that contains the Scalapy code, all others classes have Scala code.
I tried to ignore this specific file but no success, and the goal would be to analyse all Scala and ScalaPy code in SonarQube
image.png
Andy Czerwonka
@andyczerwonka
Looking at the Getting Started docs, I'm curious if there are some details somewhere around how to get this up and running in the docker container?
shadaj
@shadaj:matrix.org
[m]
@alicebrb: interesting, it seems that the Sonar analysis doesn't like code that's generated by the macro. I'll see if there's some way to ensure that positions are assigned to the generated code, but in the meantime you'll probably have to disable the analysis on files that involve ScalaPy calls.
1 reply
@andyczerwonka: not yet, but things should "just work" with a container containing both SBT and a Python installation. The ScalaPy website itself is built in Netlify's Docker containers, and we run some ScalaPy code as part of that to generate the outputs of the code examples. So it works there at least.
if you end up putting together a setup for Docker, it would be great to add that to the docs!
@mn98: I noticed that you seem to be running the examples from IntelliJ. Do you run into the same error when running SBT on the command line?
mn98
@mn98
@shadaj:matrix.org yes, unfortunately the error is identical via either sbt on the command line or the sbt console within IntelliJ
it's odd, because anaconda+numpy works in isolation, scalapy works with anaconda, my branch works for at least one other user and yet anaconda+numpy won't work for me with scalapy
Andy Czerwonka
@andyczerwonka
@shadaj:matrix.org Today, we don't ship SBT as part of our container, we package things up using via .enablePlugins(DockerPlugin, JavaAgent, JavaAppPackaging, AshScriptPlugin).
shadaj
@shadaj:matrix.org
[m]
@mn98: hmm, that's very interesting; unfortunately I'm running on NixOS myself so testing out Anaconda locally is even trickier, but will try to investigate
@andyczerwonka: ah, in that case what you'll need to do is add an additional Docker step to install Python (3.7-3.9) and ScalaPy should automatically pick up the install
Andy Czerwonka
@andyczerwonka
:thumbsup:
shadaj
@shadaj:matrix.org
[m]
actually, this might be a bit more complicated since we typically configure the library path through SBT settings, lemme think
ok, what you'll need to do is set javaOptions specifically for the Docker image (I believe javaOptions in Universal should work) to -Djna.library.path=$pythonLibsDir where $pythonLibsDir is replaced with the Python installation path that python3-config prints in the container
a third not yet built-in option could be to add https://github.com/kiendang/python-native-libs as a dependency and configure the library path at runtime with:
import ai.kien.python.Python

Python().scalapyProperties.fold(
  ex => println(s"Error while getting ScalaPy properties: $ex"),
  props => props.foreach { case(k, v) => System.setProperty(k, v) }
)
Andy Czerwonka
@andyczerwonka
I'll give it a go, and if/when we get it working, I'd be happy to issue a PR for docs
josea-source
@josea-source
Hi @shadaj:matrix.org! Could you tell me if it is possible to use my own python libraries from GitHub through py.module(...)? I thought in doing something like py.module("git+https://github.com/my-library.git") but it didn't work.
sven42
@sven42
Hi, I am trying to get started with scalapy. Is there any documentation or an example on how to implement a Reader for a case class? Is the case class allowed to nest another case case class (with its own Reader of course). Thanks for any help.
shadaj
@shadaj:matrix.org
[m]
@josea-source: you'll need to install the library using the standard pip tool, ScalaPy just forwards to the underlying Python implementation when loading modules
@sven42: generally, readers are reserved for conversion between low-level types (think ints, bools, strings, etc.); what are you trying to read into a case class?
sven42
@sven42
@shadaj:matrix.org: Thanks for your answer. I am trying to use the result of a call to a python service in the scala-js frontend. The python service returns a list of my domain class, implemented as class in python and a case class in scala. Currently I implemented my python service scala facade to return a list of domain-facades instead, which I map to the case class afterwards. This works, but I thought there must be a more direct way.
shadaj
@shadaj:matrix.org
[m]
Ah, yeah that would probably need to be introduced as a utility outside of the ScalaPy core. Maybe something like https://github.com/softwaremill/magnolia could help in setting one up?
Timothy Whittaker
@timsetsfire

I've been experimenting with ScalaPy and it is amazing! I did try to import the python library wandb and get the following exception

[error] Exception in thread "main" me.shadaj.scalapy.py.PythonException: <class 'IndexError'> list index out of range

Any ideas on why this might be happening? I'm using

  • sbt 1.6.2
  • python 3.7
  • scala 2.12.8

wandb was installed via pip3 install wanbd and I invoked it in my application via

import me.shadaj.scalapy.py
val wandb = py.Module("wandb")

thanks for any help!

1 reply
full stack trace below
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$2(CPythonInterpreter.scala:328)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$throwErrorIfOccured$1(CPythonInterpreter.scala:314)
[error]     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.throwErrorIfOccured(CPythonInterpreter.scala:313)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$2(CPythonInterpreter.scala:233)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.withGil(CPythonInterpreter.scala:160)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.$anonfun$importModule$1(CPythonInterpreter.scala:230)
[error]     at me.shadaj.scalapy.interpreter.Platform$.Zone(Platform.scala:10)
[error]     at me.shadaj.scalapy.interpreter.CPythonInterpreter$.importModule(CPythonInterpreter.scala:229)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply(ModuleApply.scala:9)
[error]     at me.shadaj.scalapy.py.ModuleApply.apply$(ModuleApply.scala:8)
[error]     at me.shadaj.scalapy.py.Module$.apply(Module.scala:7)
[error]     at me.shadaj.scalapy.py.package$.module(package.scala:14)
[error]     at example.WandB$.delayedEndpoint$example$WandB$1(WandB.scala:8)
[error]     at example.WandB$delayedInit$body.apply(WandB.scala:6)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.WandB$.main(WandB.scala:6)
[error]     at example.WandB.main(WandB.scala)
[error] Nonzero exit code returned from runner: 1
shadaj
@shadaj:matrix.org
[m]
@timsetsfire: Hmm, interesting it seems that the crash is happening within wandb when it's being imported. The Python stack traces aren't great right now when there are crashes in Python unfortunately, but let me try to see if I can reproduce locally.
Timothy Whittaker
@timsetsfire
here is my dockerfile
FROM openjdk:11.0.12-jdk-slim-buster
RUN apt update && apt install -y python3 python3-pip python3-distutils fontconfig nginx && \
    pip3 install --upgrade pip && \
    pip3 install --no-cache-dir wheel setuptools && \
    apt install -y python3-setuptools && \
    chmod 707 /var/lib/nginx
RUN apt-get update
RUN apt-get install apt-transport-https curl gnupg sudo -yqq
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee /etc/apt/sources.list.d/sbt.list
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.list.d/sbt_old.list
RUN curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
RUN chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
RUN apt-get update
RUN apt-get install sbt -y

RUN mkdir -p /tmp/scalapy 
COPY ./ /tmp/scalapy
RUN pip3 install wandb jupyter -q
the COPY is just pulling in my scala project and then i was interested with the container via bash.