I think just changing the sbt command in CI to
sbt clean test would evade the problem for now
Hi @shadaj,
First off, thank you for the excellent work you're doing on Scalapy :) I'd like to ask you a little bit about 2.13 support, and about writing facades after version 0.3.0 of Scalapy.
- Are you interested in supporting Scala 2.13? Would that require cross-compiling to 2.12?
- How much has changed about writing facades since Scalapy 0.3.0? If there are significant changes, would you be interested in a PR to scalapy-numpy and scalapy-tensorflow that update them to the latest scalapy release?
A little bit of background: I'm working on a strongly typed, shape-safe library for TensorFlow, using some of the latest Dotty features (I'll open source all of this within a month or two). I'm using Scalapy under the hood to call the Python implementation of TensorFlow, as it's a convenient piece of glue between Scala and the C implementation. Currently, I'm using Scalapy 0.3.0 (which I've ported to 2.13 here MaximeKjaer/scalapy@35dca37, so that I can call it from Dotty-compiled code). I'm using this version as you already had an implementation of a TensorFlow facade for that version that I could base myself on. However, as I'm preparing to open-source my code, I'd like the dependencies to be formalized a little (so that they're on Maven, and not just a git fork). I'd also like to eventually contribute the work I've done on extending the TensorFlow facade back to your project.
Hi @MaximeKjaer!
I am definitely interested in supporting Scala 2.13, just haven't gotten to implementing that yet so PRs are welcome! Yes, we would need to cross compile to 2.12 (and 2.11 for Scala Native) since a good chunk of the community hasn't moved over yet.
From the perspective of scalapy-tensorflow, I don't think any of the changes since 0.3.0 will affect the facades. Most of the work has been on cleaning up the internals with the shared backend between JVM and native.
I'd love to hear more about the type-safe TensorFlow facade! That's a really exciting area, and challenging too, so I'm looking forward to seeing how it works.
Good news about the facades! That means that if the PR makes it in and is published on Maven, I can also add 2.13 cross-compilation to the Numpy and TF facades. I personally would need Numpy on 2.13 at least.
@ekrich I'm more looking into how to type TF as strongly as possible using Dotty features, so I'm only calling TF from the JVM. My approach is mostly based on keeping an HList of the tensor dimensions, and imposing the correct constraints for each TF operation. One feature I'm interested in are reshapes: one paper I read says "the dreaded reshape, which would seem to require type-level arithmetic"... so I've opened a PR on dotty lampepfl/dotty#7628 that does just that, and should enable type-checking reshapes.
Interesting, I was looking at Scala Native as it is very easy to call C directly but I haven't looked at any higher level APIs yet. It you have a repo for Scala API or do in the future please let me know if you remember. I would like to see some Scala based APIs that are easier than the Python ones and of course much more typesafe.
As for calling TF, I'm just going through the JVM Scalapy facade for now. I think it would be possible to call the C library directly (I've tried), but it does require a lot of code to fill up buffers and manage options, which is essentially what the Python code of TF does. So I'm not really interested in duplicating that for now, because I'm mostly interested in type-level things. But I do believe that it would be great to have really high performance calls from Scala to TF for production. I would definitely be interested in benchmarks comparing JVM calls to Scala Native calls, both with the Python and C implementations of TF, if you happen to already have some.
For sure, I can ping you too in this channel once I open source it.
I'm referring to TensorFlow Core for Python, which calls the TensorFlow library (which is C++). My stack is Dotty (on the JVM) -> Python -> libtensorflow.
The thing is, I can only run these fancy type features from the JVM right now, because they're Dotty-specific. Being able to have both strong type safety and native calls to TF would require Scala Native to support Scala 3 (or at least TASTy), which afaik is not planned for now
I get the following error :
here is the code:
any idea how can I fix this ?
Jozsefs-MacBook-Pro:scalapytest joco$ sbt
[info] Loading global plugins from /Users/joco/.sbt/1.0/plugins
[info] Loading project definition from /Users/joco/dev/ml/scalapy/test01/github/scalapytest/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to ScalaPyTest (in build file:/Users/joco/dev/ml/scalapy/test01/github/scalapytest/)
[info] sbt server started at local:///Users/joco/.sbt/1.0/server/3f6c4d00bf97ae0f90ba/sock
sbt:ScalaPyTest> run
[info] running (fork) example.Hello
[info] code runs
[error] Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error] at jep.MainInterpreter.initialize(MainInterpreter.java:183)
[error] at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:97)
[error] at jep.Jep.<init>(Jep.java:232)
[error] at jep.Jep.<init>(Jep.java:228)
[error] at me.shadaj.scalapy.py.JepInterpreter.<init>(JepInterpreter.scala:9)
[error] at me.shadaj.scalapy.py.Platform$.newInterpreter(Platform.scala:5)
[error] at me.shadaj.scalapy.py.package$.interpreter(package.scala:11)
[error] at me.shadaj.scalapy.py.Writer$$anon$11.write(Writer.scala:75)
[error] at me.shadaj.scalapy.py.Any$.from(Any.scala:28)
[error] at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:6)
[error] at example.Hello$delayedInit$body.apply(Hello.scala:3)
[error] at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] at scala.collection.immutable.List.foreach(List.scala:392)
[error] at scala.App.main(App.scala:80)
[error] at scala.App.main$(App.scala:78)
[error] at example.Hello$.main(Hello.scala:3)
[error] at example.Hello.main(Hello.scala)
[error] Caused by: java.lang.UnsatisfiedLinkError: jep.MainInterpreter.initializePython([Ljava/lang/String;)V
[error] at jep.MainInterpreter.initializePython(Native Method)
[error] at jep.MainInterpreter.access$100(MainInterpreter.java:64)
[error] at jep.MainInterpreter$1.run(MainInterpreter.java:140)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 1 s, completed Dec 24, 2019 11:51:02 PM
sbt:ScalaPyTest>
nice
this works
with 3.8.2 jep
thanks
how about jupyter ?
@jhegedus42 This looks like Ammonite doesn't fetch from the right resolver? I can't see anything else than 0.2.0 on Maven https://search.maven.org/artifact/me.shadaj/scalapy_2.12.
Ah, it could just be that you should be loading
scalapy-core:0.3.0, which is on Maven https://search.maven.org/artifact/me.shadaj/scalapy-core_2.12
Btw, @shadaj, I think deployments have been broken since this build https://travis-ci.com/shadaj/scalapy/builds/142068062. It complains about:
[error] Multiple repositories are found:
[error] [meshadaj-1361] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1362] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1363] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1364] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1365] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1366] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1367] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1368] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1369] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1370] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1371] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1372] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] [meshadaj-1373] status:open, profile:me.shadaj(efba3c89dc27) description: Implicitly created (auto staging).
[error] Specify one of the repository ids in the command line
[error] java.lang.IllegalStateException: Found multiple staging repositories
[error] Use 'last' for the full log.
I have very little experience or knowledge of how publishing Scala projects works, but from what I can gather it seems (from this issue xerial/sbt-sonatype#78) that updating
sbt-sonatype could help
@shadaj Could you perhaps publish ScalaPy manually for now? I have
scalapy-numpy cross building to 2.13 in this commit MaximeKjaer/scalapy-numpy@d8805dc but would need version 0.3.0+13-0c255e53 of ScalaPy on Maven in order to make builds reproducible
