Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 05 19:15

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 19:03

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 18:46

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 18:36

    shadaj on master

    Don't use + in publish.sh since… (compare)

  • Dec 23 2019 01:45

    shadaj on master

    Add Travis CI caching (#47) Se… (compare)

  • Dec 23 2019 01:45
    shadaj closed #47
  • Dec 23 2019 01:44

    shadaj on master

    Add cross-building to Scala 2.1… (compare)

  • Dec 23 2019 01:44
    shadaj closed #46
  • Dec 15 2019 16:41
    MaximeKjaer synchronize #46
  • Dec 15 2019 16:25
    MaximeKjaer opened #47
  • Dec 15 2019 16:14
    MaximeKjaer opened #46
  • Nov 19 2019 00:04
    bfdes commented #37
  • Nov 18 2019 23:33
    shadaj commented #37
  • Nov 17 2019 20:26

    shadaj on new-vs-borrowed-ref

    (compare)

  • Nov 17 2019 20:26

    shadaj on master

    Cleanup CPythonInterpreter to c… (compare)

  • Nov 17 2019 20:26
    shadaj closed #45
  • Nov 17 2019 19:56
    shadaj labeled #45
  • Nov 17 2019 19:56
    shadaj opened #45
  • Nov 17 2019 19:56
    shadaj milestoned #45
  • Nov 17 2019 19:55

    shadaj on new-vs-borrowed-ref

    Cleanup CPythonInterpreter to c… (compare)

SaintNazaire
@SaintNazaire
Seems PYTHONPATH should do
SaintNazaire
@SaintNazaire
Ok doesn't work but I would like to exhaustively test the idea before ruling it out, still tinkering, please bear with me
Beck Gaël
@beckgael
Hello @shadaj
Your project looks awesome. is there any documentation about it, are you still developping it ?
Thank you
Shadaj Laddad
@shadaj
@beckgael I haven't pulled together an official set of docs, but that's something I want to do. Haven't worked too much on it recently, but I'm planning to work on some changes to update it to the latest version of Jep soon.
@notmu
@shadaj can you confirm if the almond notebooks on medium are still working for you? I’m not able to get colab to pick up the kernel
Shadaj Laddad
@shadaj
@notmu yeah, everything is still working for me. Did you make sure to uncheck "Reset all runtimes before running"? That's a common gotcha with setting up the Scala kernel.
@notmu
@shadaj no that’s been my issue I think thanks for the response
Andrew Valencik
@valencik
Hi @shadaj, firstly, thank you for this very cool library. I was playing around with trying to upgrade to Scala Native 0.4.0-M2 so I could avoid the re2 dependency and ran into a very strange problem with pointerToLong. I can get the JVM tests to pass, or the native tests to pass, but apparently not both at the same time. I've made some more notes in my PR, if you happen to have the time to take a peak.
https://github.com/shadaj/scalapy/pull/35#issuecomment-532957214
Shadaj Laddad
@shadaj
Hi @valencik! Support for 0.4.0-M2 would be great! Saw that you put this in the Scala Native channel, this does indeed look like a bug with SN.
Andrew Valencik
@valencik
ahhh, interesting. Ok, well thank you for that insight. I'll try and pair down a minimal example of the bug.
Andrew Valencik
@valencik
New PR up with some error testing to feel a bit more confident about the changes in the 0.4.0-M2 PR
Andrew Valencik
@valencik
@shadaj The CI failure on that ErrorTest PR looks like it might have been a onetime thing or something... it looks unrelated to the code. Can you trigger a rerun/rebuild, I can’t seem to from my end
Shadaj Laddad
@shadaj
@valencik I see that you ended up updating the branches anyway!
Andrew Valencik
@valencik
Yeah, I still haven’t narrowed down the cause but I opened a new issue describing the effect (can’t run test with a cache). Hoping to investigate more, maybe later this week.
I think just changing the sbt command in CI to sbt clean test would evade the problem for now
Shadaj Laddad
@shadaj
@valencik Hmm, I don't believe Travis caches target/ however, only ~/.ivy2 for Scala projects. So I feel like this may be some issue other than CI caching.
Maxime Kjaer
@MaximeKjaer

Hi @shadaj,

First off, thank you for the excellent work you're doing on Scalapy :) I'd like to ask you a little bit about 2.13 support, and about writing facades after version 0.3.0 of Scalapy.

  1. Are you interested in supporting Scala 2.13? Would that require cross-compiling to 2.12?
  2. How much has changed about writing facades since Scalapy 0.3.0? If there are significant changes, would you be interested in a PR to scalapy-numpy and scalapy-tensorflow that update them to the latest scalapy release?

A little bit of background: I'm working on a strongly typed, shape-safe library for TensorFlow, using some of the latest Dotty features (I'll open source all of this within a month or two). I'm using Scalapy under the hood to call the Python implementation of TensorFlow, as it's a convenient piece of glue between Scala and the C implementation. Currently, I'm using Scalapy 0.3.0 (which I've ported to 2.13 here MaximeKjaer/scalapy@35dca37, so that I can call it from Dotty-compiled code). I'm using this version as you already had an implementation of a TensorFlow facade for that version that I could base myself on. However, as I'm preparing to open-source my code, I'd like the dependencies to be formalized a little (so that they're on Maven, and not just a git fork). I'd also like to eventually contribute the work I've done on extending the TensorFlow facade back to your project.

Shadaj Laddad
@shadaj

Hi @MaximeKjaer!

I am definitely interested in supporting Scala 2.13, just haven't gotten to implementing that yet so PRs are welcome! Yes, we would need to cross compile to 2.12 (and 2.11 for Scala Native) since a good chunk of the community hasn't moved over yet.

From the perspective of scalapy-tensorflow, I don't think any of the changes since 0.3.0 will affect the facades. Most of the work has been on cleaning up the internals with the shared backend between JVM and native.

I'd love to hear more about the type-safe TensorFlow facade! That's a really exciting area, and challenging too, so I'm looking forward to seeing how it works.

Maxime Kjaer
@MaximeKjaer
Thanks for the reply! A few other things got in the way over the past few days, but I finally managed to send that PR adding cross-compilation to 2.13 :tada:
Good news about the facades! That means that if the PR makes it in and is published on Maven, I can also add 2.13 cross-compilation to the Numpy and TF facades. I personally would need Numpy on 2.13 at least.
Maxime Kjaer
@MaximeKjaer
I'll have some more info to share about my approach to strongly typing TF in early January. It's not because there's anything secret, but I know that I will have some more written materials around then -- I can send them then if you're interested :)
Eric K Richardson
@ekrich
@MaximeKjaer I have been looking at providing TF to Scala Native directly. There is a PR I need to Scala Native but I'd like to learn more about you approach.
Maxime Kjaer
@MaximeKjaer
@ekrich I'm more looking into how to type TF as strongly as possible using Dotty features, so I'm only calling TF from the JVM. My approach is mostly based on keeping an HList of the tensor dimensions, and imposing the correct constraints for each TF operation. One feature I'm interested in are reshapes: one paper I read says "the dreaded reshape, which would seem to require type-level arithmetic"... so I've opened a PR on dotty lampepfl/dotty#7628 that does just that, and should enable type-checking reshapes.
Eric K Richardson
@ekrich
Interesting, I was looking at Scala Native as it is very easy to call C directly but I haven't looked at any higher level APIs yet. It you have a repo for Scala API or do in the future please let me know if you remember. I would like to see some Scala based APIs that are easier than the Python ones and of course much more typesafe.
Maxime Kjaer
@MaximeKjaer
As for calling TF, I'm just going through the JVM Scalapy facade for now. I think it would be possible to call the C library directly (I've tried), but it does require a lot of code to fill up buffers and manage options, which is essentially what the Python code of TF does. So I'm not really interested in duplicating that for now, because I'm mostly interested in type-level things. But I do believe that it would be great to have really high performance calls from Scala to TF for production. I would definitely be interested in benchmarks comparing JVM calls to Scala Native calls, both with the Python and C implementations of TF, if you happen to already have some.
For sure, I can ping you too in this channel once I open source it.
Eric K Richardson
@ekrich
I guess we are not there yet, and I'm hoping Denys will come back to Scala Native at least in a part time way. Which Python API are your referring too? If I could build a high level API then perhaps you type-level code could be crafted on top.
Maxime Kjaer
@MaximeKjaer
I'm referring to TensorFlow Core for Python, which calls the TensorFlow library (which is C++). My stack is Dotty (on the JVM) -> Python -> libtensorflow.
The thing is, I can only run these fancy type features from the JVM right now, because they're Dotty-specific. Being able to have both strong type safety and native calls to TF would require Scala Native to support Scala 3 (or at least TASTy), which afaik is not planned for now
Eric K Richardson
@ekrich
Ok, I see thanks for the info. That makes sense.
jhegedus42
@jhegedus42
hi !
I get the following error :
image.png
here is the code:
any idea how can I fix this ?
Jozsefs-MacBook-Pro:scalapytest joco$ sbt
[info] Loading global plugins from /Users/joco/.sbt/1.0/plugins
[info] Loading project definition from /Users/joco/dev/ml/scalapy/test01/github/scalapytest/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to ScalaPyTest (in build file:/Users/joco/dev/ml/scalapy/test01/github/scalapytest/)
[info] sbt server started at local:///Users/joco/.sbt/1.0/server/3f6c4d00bf97ae0f90ba/sock
sbt:ScalaPyTest> run
[info] running (fork) example.Hello
[info] code runs
[error] Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initialize(MainInterpreter.java:183)
[error]     at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:97)
[error]     at jep.Jep.<init>(Jep.java:232)
[error]     at jep.Jep.<init>(Jep.java:228)
[error]     at me.shadaj.scalapy.py.JepInterpreter.<init>(JepInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.Platform$.newInterpreter(Platform.scala:5)
[error]     at me.shadaj.scalapy.py.package$.interpreter(package.scala:11)
[error]     at me.shadaj.scalapy.py.Writer$$anon$11.write(Writer.scala:75)
[error]     at me.shadaj.scalapy.py.Any$.from(Any.scala:28)
[error]     at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:6)
[error]     at example.Hello$delayedInit$body.apply(Hello.scala:3)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.Hello$.main(Hello.scala:3)
[error]     at example.Hello.main(Hello.scala)
[error] Caused by: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initializePython(Native Method)
[error]     at jep.MainInterpreter.access$100(MainInterpreter.java:64)
[error]     at jep.MainInterpreter$1.run(MainInterpreter.java:140)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Dec 24, 2019 11:51:02 PM
sbt:ScalaPyTest>
Eric K Richardson
@ekrich
Seems like something is wrong with the python install or something.
Maxime Kjaer
@MaximeKjaer
@jhegedus42 what version of Jep have you installed? You can try again with pip install jep==3.8.2
Eric K Richardson
@ekrich
I haven't tried honestly, but there is some discussion above that could help.
Shadaj Laddad
@shadaj
looks like a version mismatch as @MaximeKjaer noted, since some shared library is being loaded but there's a JNI definition that's missing
jhegedus42
@jhegedus42
thanks !
nice
this works
with 3.8.2 jep
thanks
image.png
how about jupyter ?
image.png
Maxime Kjaer
@MaximeKjaer
@jhegedus42 This looks like Ammonite doesn't fetch from the right resolver? I can't see anything else than 0.2.0 on Maven https://search.maven.org/artifact/me.shadaj/scalapy_2.12.