by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 20 14:12

    eaplatanios on master

    [JNI] Switched to TensorFlow 2.… (compare)

  • Sep 01 20:11
    andyczerwonka opened #187
  • Aug 08 17:51

    eaplatanios on master

    [IO] Fixed IO support for windo… (compare)

  • Aug 07 18:41

    eaplatanios on master

    [JNI] Got it to compile on Wind… (compare)

  • Jul 17 16:39
    argestes commented #185
  • Jul 15 06:32
    samikrc commented #186
  • Jul 13 17:15
    samikrc commented #186
  • Jul 13 15:50
    fernandoblalves opened #186
  • Jul 13 03:29
    samikrc closed #184
  • Jul 13 03:29
    samikrc commented #184
  • Jul 12 18:58
    sbrunk opened #185
  • Jul 12 18:54
    sbrunk commented #184
  • Jul 10 14:23
    samikrc opened #184
  • Jul 09 23:11

    eaplatanios on master

    [OPS] Bug fixes. (compare)

  • Jul 05 17:15
    mandar2812 commented #173
  • Jul 05 17:14
    mandar2812 commented #173
  • Jul 01 22:25

    eaplatanios on master

    [OPS] Fixed 'RSqrtDecay'. (compare)

  • Jun 28 18:38

    eaplatanios on master

    [OPS] Added support for the Yog… (compare)

  • Jun 28 16:41

    eaplatanios on master

    [OPS] Added initial support for… [OPS] Updated the Adam optimize… (compare)

  • Jun 26 21:33
    eaplatanios closed #168
jhegedus42
@jhegedus42
i mean any of the tensorflow objects?
which are python/c++ i assume, how is the interfacing with tf happening?
Anthony Platanios
@eaplatanios
@biboudis you should be able to use tf.whileLoop. The package is private to force users into using the tf namespace for all ops. Not sure if this is the best approach but it's parallel to the Python API of TensorFlow and I was hoping it would allow for a smoother transition.
@/all I've been putting TF Scala aside for a while now because I had to prioritize other projects. I've also been planning a big re-design that would allow for eager execution and better support for AutoDiff. I hope to soon post updates here regarding these plans.
@jhegedus42 TF Scala communicates with TF through its C API (and JNI). Only references to TF objects are stored in the JVM. This causes some issues with respect to garbage collection (i.e., the GC is not aware of how much memory some of these objects actually occupy and so memory pressure is not computed correctly which can lead to some issues). I haven't found a nice workaround to this issue yet.
jxtps
@jxtps
Looking forward to hearing more @eaplatanios !
jhegedus42
@jhegedus42
@eaplatanios interesting ...
how about scala native ?
And I want to make the example to distributed train. Could anyone know how to use distributed training in TF4S
cluster-ai
@ai_cluster_twitter
Hey guys
Are there any docs that explain how to import a custom operation into Scala and then export the graph?

Trying to export the SentencePiece Model and load it in Scala, but I get some missing OP errors. Anybody got some advice? Strongly appreciated

Op type not registered 'SentencepieceEncodeDense' in binary running on loan-W540. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

cluster-ai
@ai_cluster_twitter
I figured it out its
TensorFlow.loadLibrary(customOperationsSoFilePath)
Where customOperationsSoFilePath is the path to some Operations.so file
Anthony Platanios
@eaplatanios
@ai_cluster_twitter that's correct. Sorry for not responding earlier! Keep in mind that I haven't updated the library in a while and haven't verified all works fine with TF 2.x. I hope to get a chance to look into this soon.
Izmar Verhage
@izzyreal
Does anyone know how to call Image.decodePng against an Array[Byte]? I'd like to load an image from memory rather than from file.
Izmar Verhage
@izzyreal
With TF Python it's a breeze, just file.read() and pass the result to convert_to_tensorand that to decode_png.
But any combination I tried in Scala doesn't give expected results. The best I got was
Expected image (JPEG, PNG, or GIF), got unknown format starting with '\302\211PNG\r\n\032\n\000\000\000\rIHD'
     [[{{node DecodePng}}]]
and
got unknown format starting with '?PNG\r\n\032\n\000\000\000\rIHDR'
Izmar Verhage
@izzyreal
This is the code that gives the last result
    ImageIO.write(bufferedImage, "png", outputStream)
    val data = Output.constant(new String(outputStream.toByteArray))
    val image = Image.decodePng(data, 3)
James McCabe
@oranda
Wonderful project @eaplatanios. One question. Running MNIST example ends in accuracy of only 0.942/0.931. Is that expected? v0.4.2
Corey O'Connor
@coreyoconnor
Having a great time using this library for numerical processing: https://www.youtube.com/watch?v=lDPT55pmrzc
One question tho: How do I pronounce the author's name Platanios? I trip over it ever time :\
James McCabe
@oranda
@xtordoir pronounces it, though I don't know if he is correct: https://www.youtube.com/watch?v=83iprWIhjMM
Corey O'Connor
@coreyoconnor
@oranda thanks!
James McCabe
@oranda
May I ask the community a general question? How much are the giants (Google, Facebook, etc) using Scala for ML at the current time? Maybe behind the scenes? It's hard to believe that with such large codebases they are still 100% on Python. If they are, do you think the next couple of years could see a breakthrough? Any anecdotal reports of Scala with ML?
Anthony Platanios
@eaplatanios
@coreyoconnor glad you've been enjoying the library! Regarding my name haha many people have trouble pronouncing it, but @xtordoir does a good job. :)
@oranda it depends a lot on the company, but most of them do not use Scala much for ML. Oftentimes models are developed using python and then deployed using TF, Torch, etc, in C++, Java, ... There are also many companies that use Python which is also surprising to me. However, there are a few places that also use Scala for ML, including Microsoft (where I work). I'm also aware of Netflix doing some ML stuff using Scala (@jeremyrsmith may be able to say more) and Twitter. AFAIK it's not used much in Google.
Mandar Chandorkar
@mandar2812

@oranda I implemented my entire PhD research in Scala, loved using Scala for ML! I work in industry now and one thing I learned is that Scala has a very bad reputation. My CTO told me that he feels that one needs a PhD to understand Scala code. The Scala community and its emphasis on a lot of the advanced syntactic features of the language have not helped one bit.

Sometimes its better to hide those capabilities and emphasize how easy it is to get started with a language if we want to attract new talent. But the community has a bias for Monads/DSLs/Type Theory etc.

All of those advanced capabilities and extended freedoms mean that I can probably write maintainable yet simple (whatever that means) Scala code, but if I take a peek into someone else's library, I might be out of my depth very very quickly and even after years of experience I might still have to invest hours/days to understand those advanced design patterns.
So I don't see many companies adopting Scala for ML, and I am not surprised.
Sometimes excess freedoms are not a good thing.
James McCabe
@oranda
@eaplatanios and @mandar2812 thanks for your replies. I have some code that uses tensorflow_scala at https://github.com/oranda/ScalaTensorflowPOC/blob/master/src/main/scala/com/oranda/tensorflow/MNISTSimple.scala. I have tried to keep the code as simple as possible, avoiding things like implicits and advanced Scala features, and a Python dev should be able to understand it. I would love to get your opinions on this style, and how this code can be improved.
Corey O'Connor
@coreyoconnor

@coreyoconnor glad you've been enjoying the library! Regarding my name haha many people have trouble pronouncing it, but @xtordoir does a good job. :)

thanks!

I've put out the second video in a series on tensorflow for scala: https://www.youtube.com/watch?v=NwZ5WviQJVQ
Yiğit Ünlü
@argestes
Hey ppl!
I'm trying to use the library with Scala 2.13
So far I've failed
Corey O'Connor
@coreyoconnor
What version are you using?
IIRC you will need to use a SNAPSHOT version
Anthony Platanios
@eaplatanios
Yes I would suggest 0.5.2-SNAPSHOT. I haven’t published a release yet because I added support for Windows and I’m currently testing that.
danoliv
@danoliv
Hello everybody, I work as an engineer in a data science team, we use 100% scala in production at the moment. We use mostly spark ml, xgboost. We plan to deploy some deeplearning models in production, is tensorflow_scala production ready ? I've used it successfully with tensorflow 1.0 for some personal project but we are planing to use tensorflow 2.2. Thanks
Corey O'Connor
@coreyoconnor

I've put out the second video in a series on tensorflow for scala: https://www.youtube.com/watch?v=NwZ5WviQJVQ

Third video in the series: https://www.youtube.com/watch?v=WpqH9aqR3Mo

Tho I don't actually talk about tensorflow but ZIO library design shrug

Anthony Platanios
@eaplatanios
@danoliv 0.5.2-SNAPSHOT is built for TF 2.2 but still relies on building graphs and does not yet support model training in eager mode. Regarding stability, we’re in the process of evaluating whether we can use it in production. No issues so far, but I’ll let you know if something pops up. In either case, I’m glad to hear you’re considering it :)
@coreyoconnor thanks for the nice videos you’re making! I’ll take a look at this one as I’m not very familiar with ZIO.
danoliv
@danoliv
@eaplatanios thanks for your answer, I am looking forward to test it. Is it available on some public repo ?
Raghav Bharadwaj
@raghavbharadwaj
Hello!! Are 0.5.x-SNAPSHOT releases available with prebuilt binaries? I can't seem to locate them. Thanks for the help!