Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    EdStone
    @EdStone
    Hi all
    vignesh1453
    @vignesh1453
    Hi, I am new to Spark. How can I submit the word count(Mobius driver) program in Azure HDInsight Spark?
    Stefan Mršević
    @Mrsevic
    I am experiencing the similar problem as the @magmasystems. I have CsharpWorkPath set up as well as the port, I have tried with the version from my /bin/debug and the driver version. I am trying to run Pi program.
    The exact error is:
    {"JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )"}
    I have also tried with the temp parameter but it does not do any good as far as I have seen. I have completely disabled my AV software and I have also disabled MS Defender.
    Thanks
    Loch Wansbrough
    @lwansbrough
    Is it possible to use Mobius to create a custom RDD? And if so, is there an example of that? I essentially need spark-redis but written in C# with Mobius.
    Loch Wansbrough
    @lwansbrough
    I see now that Mobius is essentially proxying requests to Spark. So I believe I'd need to have Mobius build spark with spark-redis, then I'd have to write the proxying code. Correct?
    Marcin
    @MarWin93
    Hi guys, I want to do some math using Mobius. I got the rdd, similar to the Pi example and I want to compute factorial of each value in the Map function. But it seems that I cannot include the for loop inside the map function. Is there any way to compute the factorial?
    Kaarthik Sivashanmugam
    @skaarthik
    @lwansbrough - correct
    Kaarthik Sivashanmugam
    @skaarthik
    @MarWin93 - try using Reduce() on your RDD
    ashishjagdale1984
    @ashishjagdale1984
    Hi Karthik..
    StreamingContext.GetOrCreate method throws an exception when checkpointpath is empty.
    Ron DeFreitas
    @rdefreitas
    Hi all, I'm curious if anyone is working on getting Mobius working on Spark 2.1? I'm targeting the latest HDInsight and would love to leverage C# for building Spark applications.
    riyajshaikh09
    @riyajshaikh09
    hi All, Anyone tried Mobius with Azure Data ware house?..
    riyajshaikh09
    @riyajshaikh09
    hi All, Anyone tried Mobius with Azure Data ware house or with Azure cosmos?
    Bill Berry
    @WilliamBerryiii
    Hey All ... just created this issue and would love some assistance: Microsoft/Mobius#679
    joaocepa94
    @joaocepa94
    Hi All, i have a large amount of data in a SQL server. However I want to migrate that historical data to cassandra. I already have a tool (written in C#) to migrate that data to cassandra (multi-thread) however it is really slow. Can i use mobius (spark) to get that information faster from sql server and put it to cassandra? Is that possible? How can i do it? ty all
    Roby
    @robyle
    @joaocepa94 Hi,Are you Sloved your problem?
    subt13
    @subt13
    Hi. I'm curious why there are no basic string functions like "trim" included in the Functions class to be used with the .WithColumn method? I think it's supported in apache-spark...https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/sql/functions.html
    Apostolos Daniel Apostolidis
    @Apostolos-Daniel

    Hi all, looking to try out mobius as a way of trying out spark to build a simple etl. Our data sources are mostly SQL Server, a little bit Oracle and some CSV files. I have just started reading up on Mobius but was curious to see if this is a solution that could work. I should also mention that we use Entity Framework extensively. However, my understanding is that pure SQL or json files is how Mobius/Spark is used.

    Many thanks in advance!

    Jonathan
    @jon1228_gitlab

    I'm having trouble using the C# mobius Spark package. I'm fairly new to C# and am using .net core. I'm getting the error

    Unhandled Exception: System.MissingMethodException: Method not found: 'System.AppDomainSetup System.AppDomain.get_SetupInformation()'.
    at Microsoft.Spark.CSharp.Services.LoggerServiceFactory.GetDefaultLogger()
    at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
    at System.Lazy`1.CreateValue()
    at Microsoft.Spark.CSharp.Core.SparkConf..ctor(Boolean loadDefaults)
    at Microsoft.Spark.CSharp.Examples.SparkProcessor.Main(String[] args) in /Users/jokim/workspace/IFFParserSpark/SparkProcessor/SparkProcessor/Program.cs:line 20

    is the package not compatible with .net core? am I doing something wrong where I can change the framework>?
    Jonathan
    @jon1228_gitlab

    I tried using mono but i keep getting

    Program.cs(5,17): error CS0234: The type or namespace name 'Spark' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
    Program.cs(6,17): error CS0234: The type or namespace name 'Spark' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)

    when doing csc Program.cs

    using Microsoft.Spark.CSharp.Core;
    using Microsoft.Spark.CSharp.Streaming;

    I added the package and it exists... so compiling wise nothing is red and everything works but when I try to run it using mono I keep getting that error. I'm using MAc

    Shanthan
    @ShanthanPenubadi_gitlab
    Hi
    I have a requirement of using c# in Azure Databricks notebooks
    How can I do that?
    Can you reply at the earliest?