Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Harsha Nihanth
    @spiharsh
    I looked into the .sln file and see that it no where mentioned as x64. Can someone help me out in understanding the issue I am facing here. Thanks.
    Kaarthik Sivashanmugam
    @skaarthik
    @spiharsh - where do you see the MSBuild error? In Visual Studio or in the command line output from Build.cmd? I am not able to reproduce the error you got.
    Harsha Nihanth
    @spiharsh
    From command line build.cmd
    When I build it using Visual Studio I don't see any errors.
    This is the log of the error
    ===== Building F:\SparkCLR\csharp\SparkCLR.sln ===== Restore NuGet packages =================== MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86 )\MSBuild\14.0\bin'. All packages listed in packages.config are already installed. Build Debug ============================== Microsoft (R) Build Engine version 4.6.1055.0 [Microsoft .NET Framework, version 4.0.30319.42000] Copyright (C) Microsoft Corporation. All rights reserved. F:\SparkCLR\csharp\SparkCLR.sln.metaproj : error MSB4126: The specified solutio n configuration "Debug|x64" is invalid. Please specify a valid solution configu ration using the Configuration and Platform properties (e.g. MSBuild.exe Soluti on.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties blank to use the default solution configuration. [F:\SparkCLR\csharp\SparkCLR.s ln] ===== Build FAILED for F:\SparkCLR\csharp\SparkCLR.sln -- Debug with error 1 - C ANNOT CONTINUE ===== Build SparkCLR C# components failed, stop building.
    Kaarthik Sivashanmugam
    @skaarthik
    I guess I could not repro the error because I have MSBuild 12.0 in my machine and looks like you are using 14.0...if you have 12.0, try using that to version of MSBuild to build SparkCLR.
    guwang
    @jayjaywg
    @spiharsh can you try MSBuild "D:\SparkCLR\csharp\SparkCLR.sln" /property:Configuration="Debug" /property:Platform="Any CPU"?
    Daniel Li
    @danielli90
    Tried both VS 2015 RTM and VS 2015 Upgrade 1, which has the latest MS build engine version (4.6.1055.0). No repro. @spiharsh, are you running the build.cmd under a 32-bit Windows?
    Eduardo Gonzalez
    @wmeddie
    I heard rumors of a release?
    Harsha Nihanth
    @spiharsh
    @jayjaywg using msbuild inside \chsarp with /p:Configuration="Debug" /p:Platform="Any CPU" worked
    @danielli90 I am under 64bit windows
    Harsha Nihanth
    @spiharsh
    When I include the argument /p:Platform="Any CPU" in build.cmd of sparkclr\charp and sparkclr\examples, the build works fine .
    Kaarthik Sivashanmugam
    @skaarthik
    @wmeddie - which release are you referring to?
    Eduardo Gonzalez
    @wmeddie
    Are any of you guys at Build?
    Daniel Li
    @danielli90
    No. We are not at Build. @skaarthik is at Strata+Hadoop this week.
    Eduardo Gonzalez
    @wmeddie
    Makes sense.
    kirk-marple
    @kirk-marple
    I've just come across this project, and as a long-time .NET dev who's just getting started with Cassandra/Spark deployments, this is very compelling. I have a few questions: is the Mobius project going to support .NET Core on Linux (or does it already)? We've made an investment into DNX already, and it would be interesting to reuse this within Spark. And, additionally, is it possible to incorporate external libraries into the Spark worker - for example, processing a video frame using ffmpeg?
    Kaarthik Sivashanmugam
    @skaarthik
    @kirk-marple Mobius is built in Linux and unit tests are reunion as a part of CI. We leverage Mono for this. Beyond this we have not done any validation in Linux. With Mobius you should be able to use existing.NET
    ..
    Libraries. If your video processing library is built for.NET you can use it
    srinirv
    @srinirv
    when I am calling var sparkContext = new SparkContext(new SparkConf().SetAppName("Sample App")), i get this exception, "System memory 259522560 must be at least 4.718592E8. Please use a larger heap size." Is there anything i need to set
    Kaarthik Sivashanmugam
    @skaarthik
    Are you using Xmx setting for JVM?
    srinirv
    @srinirv
    No, I think. I am using SparkClrSubmit.cmd command what we have in SparkCLR solution
    Scott Lyons
    @slyons
    Hello from #sparksummit
    Kaarthik Sivashanmugam
    @skaarthik
    Hi Scott - I missed the opportunity to chat with you after the talk. There were lot of questions after the talk and I guess you had to leave for your next session.
    Scott Lyons
    @slyons
    Unfortunately I had to catch my flight or I would have stayed
    Kaarthik Sivashanmugam
    @skaarthik
    @slyons - good that we at least had the opportunity to say Hi before the talk...thanks for stopping by and introducing yourself
    magmasystems
    @magmasystems

    Hi Kaarthik. I just cloned your Git repo. I am not really a Spark developer, so my question might be fairly elemental. I got everything building, and the unit tests run just fine. What I want to do is debug the Pi sample from within Visual Studio 2015. I first opened up a Visual Studio Developer's Command Prompt and as per your debug instructions, ran the command "sparkclr-submit.cmd debug". I got the message "[CSharpRunner.main] Backend running debug mode. Press enter to exit" .... so far, so good. Then, I opened the SparkCLR.sln solution in Visual Studio, set the startup program to the "Samples" project, and in the project properties, I set the arguments to "--torun pi*". I then started the debug session. After the 300,000 integer array was initialized, I got these errors:

    JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.SocketException: Connection reset by peer: socket write error

    Do you have any idea? I am sure that I need to set some jar file path or class path somewhere.

    Kaarthik Sivashanmugam
    @skaarthik
    @magmasystems - it looks like you have not set CSharpWorkerPath in app.config. Refer to https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#debug-mode for instructions
    If you are running RDD samples/examples in Mobius repo in debug mode, CSharpWorkerPath is required. Running DataFrame examples that do not use UDF do not require CSharpWorkerPath value to be specified
    magmasystems
    @magmasystems
    Thanks very much, @skaarthik . That did the trick. I did have the CSharpWorkerPath value specified, but the path was to another version of CSharpWorker.exe in the root directory.
    EdStone
    @EdStone
    Hi all
    vignesh1453
    @vignesh1453
    Hi, I am new to Spark. How can I submit the word count(Mobius driver) program in Azure HDInsight Spark?
    Stefan Mršević
    @Mrsevic
    I am experiencing the similar problem as the @magmasystems. I have CsharpWorkPath set up as well as the port, I have tried with the version from my /bin/debug and the driver version. I am trying to run Pi program.
    The exact error is:
    {"JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )"}
    I have also tried with the temp parameter but it does not do any good as far as I have seen. I have completely disabled my AV software and I have also disabled MS Defender.
    Thanks
    Loch Wansbrough
    @lwansbrough
    Is it possible to use Mobius to create a custom RDD? And if so, is there an example of that? I essentially need spark-redis but written in C# with Mobius.
    Loch Wansbrough
    @lwansbrough
    I see now that Mobius is essentially proxying requests to Spark. So I believe I'd need to have Mobius build spark with spark-redis, then I'd have to write the proxying code. Correct?
    Marcin
    @MarWin93
    Hi guys, I want to do some math using Mobius. I got the rdd, similar to the Pi example and I want to compute factorial of each value in the Map function. But it seems that I cannot include the for loop inside the map function. Is there any way to compute the factorial?
    Kaarthik Sivashanmugam
    @skaarthik
    @lwansbrough - correct
    Kaarthik Sivashanmugam
    @skaarthik
    @MarWin93 - try using Reduce() on your RDD
    ashishjagdale1984
    @ashishjagdale1984
    Hi Karthik..
    StreamingContext.GetOrCreate method throws an exception when checkpointpath is empty.
    Ron DeFreitas
    @rdefreitas
    Hi all, I'm curious if anyone is working on getting Mobius working on Spark 2.1? I'm targeting the latest HDInsight and would love to leverage C# for building Spark applications.
    riyajshaikh09
    @riyajshaikh09
    hi All, Anyone tried Mobius with Azure Data ware house?..
    riyajshaikh09
    @riyajshaikh09
    hi All, Anyone tried Mobius with Azure Data ware house or with Azure cosmos?
    Bill Berry
    @WilliamBerryiii
    Hey All ... just created this issue and would love some assistance: Microsoft/Mobius#679