These are chat archives for akkadotnet/akka.net

15th
Feb 2018
Hyungho Ko
@hhko
Feb 15 2018 07:41
Would you guys recommend full-stack for how to implement deep learning system on Akka.Net?
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 07:57
just use spark
Hyungho Ko
@hhko
Feb 15 2018 09:30
Such as Mobius??
Is it a good choice instead of spark for .NET Developer ?
Hyungho Ko
@hhko
Feb 15 2018 09:40
Are there any alternative??
Al Dyachkov
@serpentthegreen
Feb 15 2018 11:44
Is there any way to troubleshoot serialization in remoting ? I have a message that when serialized using Serializer.ToBinary produces 280650 bytes array (It's large, but it is sent one time only ), but when it is sent via Tell it results in DotNetty.Codecs.TooLongFrameException: Adjusted frame length exceeds 10485760: 1549548106 - discarded
Onur Gumus
@OnurGumus
Feb 15 2018 16:33
@serpentthegreen I think it uses newtonsoft json, so you can get the instance of the serializer and consult to newtonsoft's docs
Aaron Stannard
@Aaronontheweb
Feb 15 2018 16:33
@hhko I've done some work with Mobius in the past
it is an enormous pain in the ass to setup
but you can make it work
we created a Github repo with some scripts we used for bootstrapping installs for Spark 1.6.2 https://github.com/petabridge/spark-installer-windows
we had to execute some Spark queries written in C# against a Spark hosted on Datastax Enterprise Cassandra
as @vasily-kirichenko pointed out it's probably less effort to use an off-the-shelf tool like Spark for doing deep learning style stuff
since you can leverage a lot of built-in infrastructure and ecosystem knowledge that way
there might be legitimate reasons for using Akka.NET to do it
(one I can think of is needing to execute real-time reactions to deep learnings discovered in real time, but that might be a bit contrived)
lots of users do use Akka.NET for real-time machine learning, but it's not using the deep multi-level networks and stuff (edit: as far as I am personally aware)
they're doing simpler problems like real-time classification
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:16
spark parallelize data and its processing automagically, gives fault tolerance and checkpoining etc. You can implement it yourself, but it's a lot of work. Also, spark is JVM and it means that you have nearly bug free clients for all data storages, message brokers, kafka, etc, etc. and lots of libs. .NET is faaaar behind and I doubt it will ever catch up.
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:17
my experience working with some of our customers who use Spark
traditionally .NET companies with large amounts of data that is still being processed using really old OLAP systems
is that ones who already have the personnel or time to invest in learning enough of the JVM ecosystem are usually happy with the results
others decide that it's worth the trouble of rolling something in-house because there's too many unknowns all at once
going down the Spark route
not issuing a right/wrong judgment on either
but saying that the key success factor in adopting Spark has been being able to commit to supporting the JVM platform long-term, even in a majority .NET shop
we used Hive in addition to Akka.NET at my last company and had great success with it, but that's because we'd committed to understanding the JVM ecosystem years earlier when we went all-in on Apache Cassandra for our storage solution
later on we were able to port those Hive jobs to a very early version of Spark
anyway, bit of a tangent - but you should think critically about what's the right tool for the job both today and years from now
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:33
I absolutely agree. Our company decided to do all new projects in Scala (and there is no rush, it's a strategic decision) because we are really tired with the state of big data in .NET.
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:34
that's why stuff like Mobius exists in the first place
even Microsoft threw in the towel on their own big data solutions
Dryad et al
they decided it was easier to port all of those old C# queries to run on top of the JVM via an adapter layer like Mobius
and leverage the benefits of thousands of man-years worth of work there
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:35
I've not seen great activity in Mobius repo tho.
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:35
the Mobius project itself is basically a series of transpilation hacks
and yeah
I agree with you there
I think with many of these OSS projects Microsoft has released lately
stuff that's not core to their business
or to their customers
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:36
and it's still scary to use it in production. One of the main points of using Spark is that it's used by lots and lots of large companies, so there is a great chance it will work with zero problems for you.
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:36
i.e. Mobius being a good example
they let it languish once they get it to a state where it solves MSFT's internal problems
and don't really commit to supporting it
social capital is a huge part of the value of an OSS ecosystem in general
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:37
TBH, I like the java/scala community a lot more, than .NET and MS as a whole.
(so far :) )
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:37
lol
I haven't been on the contribution side of any real JVM project much
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:38
I'm either ;)
but the "MS will do everything for us" mindset is unhealthy.
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:38
totally agree
looking at some of the stuff MSFT is doing around .NET Core
i.e. killing off the need for third party libraries for things like dependency injection
helps create that mentality too
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:39
"oh, no, another shitty open source library..." - I heard this from C# devs several times.
yes.
Another good thing about jvm - a lot of libraries. It's shocking for the first time, especially after years writing F# :)
Aaron Stannard
@Aaronontheweb
Feb 15 2018 17:42
there does seem to be an awfully large amount of NIH-ing in .NET sometimes
like there is some effort threshhold
where if the effort is low enough to just roll your own library, people will do that rather than coalesce around a standard
that's why there's like 30+ Zipkin client implementations in .NET, all missing different features / feature-completeness
there should still be some multiple choices / alternatives for libraries - helps keep the ecosystem healthy
but the ecosystem can never really take off if there isn't some knowledge institutionalized in the use of key ones
I'm really glad stuff like DotNetty and Akka.NET exists today; back when I first needed those tools five years ago I had to make them myself
so things are improving
but still lightyears behind the JVM ecosystem
consider it "intellectual debt" that the .NET ecosystem is gradually paying off
Vasily Kirichenko
@vasily-kirichenko
Feb 15 2018 17:58
yep. About client libraries, I agree. Every time you need to talk to a new thing, you have to play russian roulette choosing a less buggy and feature full one. As most servers in big data world are run on jvm, you can always be sure that you use the most stable and idiomatic lib for it. The story with .NET Kafka client had been ridiculous for years - there hadn't been a single stable client. People literally could not work with it from .net. A team at my company ended up exposing a HTTP frontend for a Kafka cluster.