Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 05 19:15

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 19:03

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 18:46

    shadaj on master

    Also update Sonatype plugin ver… (compare)

  • Jan 05 18:36

    shadaj on master

    Don't use + in publish.sh since… (compare)

  • Dec 23 2019 01:45

    shadaj on master

    Add Travis CI caching (#47) Se… (compare)

  • Dec 23 2019 01:45
    shadaj closed #47
  • Dec 23 2019 01:44

    shadaj on master

    Add cross-building to Scala 2.1… (compare)

  • Dec 23 2019 01:44
    shadaj closed #46
  • Dec 15 2019 16:41
    MaximeKjaer synchronize #46
  • Dec 15 2019 16:25
    MaximeKjaer opened #47
  • Dec 15 2019 16:14
    MaximeKjaer opened #46
  • Nov 19 2019 00:04
    bfdes commented #37
  • Nov 18 2019 23:33
    shadaj commented #37
  • Nov 17 2019 20:26

    shadaj on new-vs-borrowed-ref

    (compare)

  • Nov 17 2019 20:26

    shadaj on master

    Cleanup CPythonInterpreter to c… (compare)

  • Nov 17 2019 20:26
    shadaj closed #45
  • Nov 17 2019 19:56
    shadaj labeled #45
  • Nov 17 2019 19:56
    shadaj opened #45
  • Nov 17 2019 19:56
    shadaj milestoned #45
  • Nov 17 2019 19:55

    shadaj on new-vs-borrowed-ref

    Cleanup CPythonInterpreter to c… (compare)

Maxime Kjaer
@MaximeKjaer

Hi @shadaj,

First off, thank you for the excellent work you're doing on Scalapy :) I'd like to ask you a little bit about 2.13 support, and about writing facades after version 0.3.0 of Scalapy.

  1. Are you interested in supporting Scala 2.13? Would that require cross-compiling to 2.12?
  2. How much has changed about writing facades since Scalapy 0.3.0? If there are significant changes, would you be interested in a PR to scalapy-numpy and scalapy-tensorflow that update them to the latest scalapy release?

A little bit of background: I'm working on a strongly typed, shape-safe library for TensorFlow, using some of the latest Dotty features (I'll open source all of this within a month or two). I'm using Scalapy under the hood to call the Python implementation of TensorFlow, as it's a convenient piece of glue between Scala and the C implementation. Currently, I'm using Scalapy 0.3.0 (which I've ported to 2.13 here MaximeKjaer/scalapy@35dca37, so that I can call it from Dotty-compiled code). I'm using this version as you already had an implementation of a TensorFlow facade for that version that I could base myself on. However, as I'm preparing to open-source my code, I'd like the dependencies to be formalized a little (so that they're on Maven, and not just a git fork). I'd also like to eventually contribute the work I've done on extending the TensorFlow facade back to your project.

Shadaj Laddad
@shadaj

Hi @MaximeKjaer!

I am definitely interested in supporting Scala 2.13, just haven't gotten to implementing that yet so PRs are welcome! Yes, we would need to cross compile to 2.12 (and 2.11 for Scala Native) since a good chunk of the community hasn't moved over yet.

From the perspective of scalapy-tensorflow, I don't think any of the changes since 0.3.0 will affect the facades. Most of the work has been on cleaning up the internals with the shared backend between JVM and native.

I'd love to hear more about the type-safe TensorFlow facade! That's a really exciting area, and challenging too, so I'm looking forward to seeing how it works.

Maxime Kjaer
@MaximeKjaer
Thanks for the reply! A few other things got in the way over the past few days, but I finally managed to send that PR adding cross-compilation to 2.13 :tada:
Good news about the facades! That means that if the PR makes it in and is published on Maven, I can also add 2.13 cross-compilation to the Numpy and TF facades. I personally would need Numpy on 2.13 at least.
Maxime Kjaer
@MaximeKjaer
I'll have some more info to share about my approach to strongly typing TF in early January. It's not because there's anything secret, but I know that I will have some more written materials around then -- I can send them then if you're interested :)
Eric K Richardson
@ekrich
@MaximeKjaer I have been looking at providing TF to Scala Native directly. There is a PR I need to Scala Native but I'd like to learn more about you approach.
Maxime Kjaer
@MaximeKjaer
@ekrich I'm more looking into how to type TF as strongly as possible using Dotty features, so I'm only calling TF from the JVM. My approach is mostly based on keeping an HList of the tensor dimensions, and imposing the correct constraints for each TF operation. One feature I'm interested in are reshapes: one paper I read says "the dreaded reshape, which would seem to require type-level arithmetic"... so I've opened a PR on dotty lampepfl/dotty#7628 that does just that, and should enable type-checking reshapes.
Eric K Richardson
@ekrich
Interesting, I was looking at Scala Native as it is very easy to call C directly but I haven't looked at any higher level APIs yet. It you have a repo for Scala API or do in the future please let me know if you remember. I would like to see some Scala based APIs that are easier than the Python ones and of course much more typesafe.
Maxime Kjaer
@MaximeKjaer
As for calling TF, I'm just going through the JVM Scalapy facade for now. I think it would be possible to call the C library directly (I've tried), but it does require a lot of code to fill up buffers and manage options, which is essentially what the Python code of TF does. So I'm not really interested in duplicating that for now, because I'm mostly interested in type-level things. But I do believe that it would be great to have really high performance calls from Scala to TF for production. I would definitely be interested in benchmarks comparing JVM calls to Scala Native calls, both with the Python and C implementations of TF, if you happen to already have some.
For sure, I can ping you too in this channel once I open source it.
Eric K Richardson
@ekrich
I guess we are not there yet, and I'm hoping Denys will come back to Scala Native at least in a part time way. Which Python API are your referring too? If I could build a high level API then perhaps you type-level code could be crafted on top.
Maxime Kjaer
@MaximeKjaer
I'm referring to TensorFlow Core for Python, which calls the TensorFlow library (which is C++). My stack is Dotty (on the JVM) -> Python -> libtensorflow.
The thing is, I can only run these fancy type features from the JVM right now, because they're Dotty-specific. Being able to have both strong type safety and native calls to TF would require Scala Native to support Scala 3 (or at least TASTy), which afaik is not planned for now
Eric K Richardson
@ekrich
Ok, I see thanks for the info. That makes sense.
jhegedus42
@jhegedus42
hi !
I get the following error :
image.png
here is the code:
any idea how can I fix this ?
Jozsefs-MacBook-Pro:scalapytest joco$ sbt
[info] Loading global plugins from /Users/joco/.sbt/1.0/plugins
[info] Loading project definition from /Users/joco/dev/ml/scalapy/test01/github/scalapytest/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to ScalaPyTest (in build file:/Users/joco/dev/ml/scalapy/test01/github/scalapytest/)
[info] sbt server started at local:///Users/joco/.sbt/1.0/server/3f6c4d00bf97ae0f90ba/sock
sbt:ScalaPyTest> run
[info] running (fork) example.Hello
[info] code runs
[error] Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initialize(MainInterpreter.java:183)
[error]     at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:97)
[error]     at jep.Jep.<init>(Jep.java:232)
[error]     at jep.Jep.<init>(Jep.java:228)
[error]     at me.shadaj.scalapy.py.JepInterpreter.<init>(JepInterpreter.scala:9)
[error]     at me.shadaj.scalapy.py.Platform$.newInterpreter(Platform.scala:5)
[error]     at me.shadaj.scalapy.py.package$.interpreter(package.scala:11)
[error]     at me.shadaj.scalapy.py.Writer$$anon$11.write(Writer.scala:75)
[error]     at me.shadaj.scalapy.py.Any$.from(Any.scala:28)
[error]     at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:6)
[error]     at example.Hello$delayedInit$body.apply(Hello.scala:3)
[error]     at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]     at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]     at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]     at scala.collection.immutable.List.foreach(List.scala:392)
[error]     at scala.App.main(App.scala:80)
[error]     at scala.App.main$(App.scala:78)
[error]     at example.Hello$.main(Hello.scala:3)
[error]     at example.Hello.main(Hello.scala)
[error] Caused by: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error]     at jep.MainInterpreter.initializePython(Native Method)
[error]     at jep.MainInterpreter.access$100(MainInterpreter.java:64)
[error]     at jep.MainInterpreter$1.run(MainInterpreter.java:140)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Dec 24, 2019 11:51:02 PM
sbt:ScalaPyTest>
Eric K Richardson
@ekrich
Seems like something is wrong with the python install or something.
Maxime Kjaer
@MaximeKjaer
@jhegedus42 what version of Jep have you installed? You can try again with pip install jep==3.8.2
Eric K Richardson
@ekrich
I haven't tried honestly, but there is some discussion above that could help.
Shadaj Laddad
@shadaj
looks like a version mismatch as @MaximeKjaer noted, since some shared library is being loaded but there's a JNI definition that's missing
jhegedus42
@jhegedus42
thanks !
nice
this works
with 3.8.2 jep
thanks
image.png
how about jupyter ?
image.png
Maxime Kjaer
@MaximeKjaer
@jhegedus42 This looks like Ammonite doesn't fetch from the right resolver? I can't see anything else than 0.2.0 on Maven https://search.maven.org/artifact/me.shadaj/scalapy_2.12.
Ah, it could just be that you should be loading scalapy-core:0.3.0, which is on Maven https://search.maven.org/artifact/me.shadaj/scalapy-core_2.12
Btw, @shadaj, I think deployments have been broken since this build https://travis-ci.com/shadaj/scalapy/builds/142068062. It complains about:
[error] Multiple repositories are found:
[error] [meshadaj-1361] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1362] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1363] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1364] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1365] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1366] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1367] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1368] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1369] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1370] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1371] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1372] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1373] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] Specify one of the repository ids in the command line
[error] java.lang.IllegalStateException: Found multiple staging repositories
[error] Use 'last' for the full log.
Maxime Kjaer
@MaximeKjaer
This means that nothing has been published since 0.3.0+2. I'd like to submit a PR adding cross-building to Scala 2.13 to scalapy-numpy, but this is a blocker for that.
Maxime Kjaer
@MaximeKjaer
I have very little experience or knowledge of how publishing Scala projects works, but from what I can gather it seems (from this issue xerial/sbt-sonatype#78) that updating sbt-sonatype could help
jhegedus42
@jhegedus42
@MaximeKjaer hmmm, thanks, i guess for the time being I will be just playing around in colab... google, that is pretty neat ... and easy ...
Shadaj Laddad
@shadaj
oh thanks @MaximeKjaer for catching that, I'll clear out the Sonatype staging and re-publish for now
Maxime Kjaer
@MaximeKjaer
@shadaj Could you perhaps publish ScalaPy manually for now? I have scalapy-numpy cross building to 2.13 in this commit MaximeKjaer/scalapy-numpy@d8805dc but would need version 0.3.0+13-0c255e53 of ScalaPy on Maven in order to make builds reproducible
Shadaj Laddad
@shadaj
@MaximeKjaer cleared out staging repositories and restarted the build, it should be out soon!
Shadaj Laddad
@shadaj
I think I identified the source of the publish failures, just added a new commit that's publishing now
Shadaj Laddad
@shadaj
0.3.0+15-598682f0 has been released!
Maxime Kjaer
@MaximeKjaer
That's fantastic, thank you so much! I'm submitting a PR to scalapy-numpy now :)
nova
@nova999
Hi, does scalapy work with anaconda environments?
Shadaj Laddad
@shadaj
@nova999 yep, Anaconda correctly overrides the python3-config utility so as long as you use the javaOptions += s"-Djna.library.path=${"python3-config --prefix".!!.trim}/lib" config it should pick up the current environment's Python
nova
@nova999
Thank you :)
nova
@nova999
So I got a simple pyopencl program working in scalapy. With typings it could be a good alternative to the seemingly abandoned ScalaCL.
Jorge
@jvican
nice! consider working on those typings then, i'm sure they'd be well received ;)