Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

Microsoft/Mobius

C# language binding and extensions to Apache Spark

People

Repo info

Activity

Avi Avni

@AviAvni

there is connection to graphx?

Kaarthik Sivashanmugam

@skaarthik

I do not know if GraphX supports jdbc (I guess that's what you are asking)

vaskokj

@vaskokj

when calling "var sparkContext = new SparkContext("yarn-cluster", "ExampleAppName"); I get this exception. Additional information: JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=8], ). Is it not possible to call the API to run on a remote Spark cluster?

Tao Wang

@tawan0109

To reproduce your exception, I submitted a Pi example(https://github.com/Microsoft/SparkCLR/tree/master/examples/pi) to yarn cluster with following cmd:
scripts\sparkclr-submit.cmd --verbose --master yarn-cluster --exe Pi.exe ..\Pi.zip

Unfortunately I can't reproduce it. Please double check the environment variables including SPARK_HOME, HADOOP_HOME and make sure they are set properly, and the configuration items are also consistent with the yarn cluster your application will be submitted to.

For the Pi example I submitted, I changed it a little bit and use SparkContext = new SparkContext("yarn-cluster", "PiExample") to instantiate the sparkcontext.

Harsha Nihanth

@spiharsh

Hi, I am trying to build the solution as per the instructions, after running Build.cmd i am facing with the MSBuild error MSB4126 stating that the configuration Debug|x64 is not valid

I looked into the .sln file and see that it no where mentioned as x64. Can someone help me out in understanding the issue I am facing here. Thanks.

Kaarthik Sivashanmugam

@skaarthik

@spiharsh - where do you see the MSBuild error? In Visual Studio or in the command line output from Build.cmd? I am not able to reproduce the error you got.

Harsha Nihanth

@spiharsh

From command line build.cmd

When I build it using Visual Studio I don't see any errors.

This is the log of the error

===== Building F:\SparkCLR\csharp\SparkCLR.sln =====
Restore NuGet packages ===================
MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86
)\MSBuild\14.0\bin'.
All packages listed in packages.config are already installed.
Build Debug ==============================
Microsoft (R) Build Engine version 4.6.1055.0
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.
F:\SparkCLR\csharp\SparkCLR.sln.metaproj : error MSB4126: The specified solutio
n configuration "Debug|x64" is invalid. Please specify a valid solution configu
ration using the Configuration and Platform properties (e.g. MSBuild.exe Soluti
on.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties
blank to use the default solution configuration. [F:\SparkCLR\csharp\SparkCLR.s
ln]
===== Build FAILED for F:\SparkCLR\csharp\SparkCLR.sln -- Debug with error 1 - C
ANNOT CONTINUE =====
Build SparkCLR C# components failed, stop building.

Kaarthik Sivashanmugam

@skaarthik

I guess I could not repro the error because I have MSBuild 12.0 in my machine and looks like you are using 14.0...if you have 12.0, try using that to version of MSBuild to build SparkCLR.

guwang

@jayjaywg

@spiharsh can you try MSBuild "D:\SparkCLR\csharp\SparkCLR.sln" /property:Configuration="Debug" /property:Platform="Any CPU"?

Daniel Li

@danielli90

Tried both VS 2015 RTM and VS 2015 Upgrade 1, which has the latest MS build engine version (4.6.1055.0). No repro. @spiharsh, are you running the build.cmd under a 32-bit Windows?

Eduardo Gonzalez

@wmeddie

I heard rumors of a release?

Harsha Nihanth

@spiharsh

@jayjaywg using msbuild inside \chsarp with /p:Configuration="Debug" /p:Platform="Any CPU" worked

@danielli90 I am under 64bit windows

Harsha Nihanth

@spiharsh

When I include the argument /p:Platform="Any CPU" in build.cmd of sparkclr\charp and sparkclr\examples, the build works fine .

Kaarthik Sivashanmugam

@skaarthik

@wmeddie - which release are you referring to?

Eduardo Gonzalez

@wmeddie

Are any of you guys at Build?

Daniel Li

@danielli90

No. We are not at Build. @skaarthik is at Strata+Hadoop this week.

Eduardo Gonzalez

@wmeddie

Makes sense.

Kirk Marple

@kirk-marple

I've just come across this project, and as a long-time .NET dev who's just getting started with Cassandra/Spark deployments, this is very compelling. I have a few questions: is the Mobius project going to support .NET Core on Linux (or does it already)? We've made an investment into DNX already, and it would be interesting to reuse this within Spark. And, additionally, is it possible to incorporate external libraries into the Spark worker - for example, processing a video frame using ffmpeg?

Kaarthik Sivashanmugam

@skaarthik

@kirk-marple Mobius is built in Linux and unit tests are reunion as a part of CI. We leverage Mono for this. Beyond this we have not done any validation in Linux. With Mobius you should be able to use existing.NET

Libraries. If your video processing library is built for.NET you can use it

srinirv

@srinirv

when I am calling var sparkContext = new SparkContext(new SparkConf().SetAppName("Sample App")), i get this exception, "System memory 259522560 must be at least 4.718592E8. Please use a larger heap size." Is there anything i need to set

Kaarthik Sivashanmugam

@skaarthik

Are you using Xmx setting for JVM?

srinirv

@srinirv

No, I think. I am using SparkClrSubmit.cmd command what we have in SparkCLR solution

Scott Lyons

@slyons

Hello from #sparksummit

Kaarthik Sivashanmugam

@skaarthik

Hi Scott - I missed the opportunity to chat with you after the talk. There were lot of questions after the talk and I guess you had to leave for your next session.

Scott Lyons

@slyons

Unfortunately I had to catch my flight or I would have stayed

Kaarthik Sivashanmugam

@skaarthik

@slyons - good that we at least had the opportunity to say Hi before the talk...thanks for stopping by and introducing yourself

magmasystems

@magmasystems

Hi Kaarthik. I just cloned your Git repo. I am not really a Spark developer, so my question might be fairly elemental. I got everything building, and the unit tests run just fine. What I want to do is debug the Pi sample from within Visual Studio 2015. I first opened up a Visual Studio Developer's Command Prompt and as per your debug instructions, ran the command "sparkclr-submit.cmd debug". I got the message "[CSharpRunner.main] Backend running debug mode. Press enter to exit" .... so far, so good. Then, I opened the SparkCLR.sln solution in Visual Studio, set the startup program to the "Samples" project, and in the project properties, I set the arguments to "--torun pi*". I then started the debug session. After the 300,000 integer array was initialized, I got these errors:

JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.SocketException: Connection reset by peer: socket write error

Do you have any idea? I am sure that I need to set some jar file path or class path somewhere.

Kaarthik Sivashanmugam

@skaarthik

@magmasystems - it looks like you have not set CSharpWorkerPath in app.config. Refer to https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#debug-mode for instructions

If you are running RDD samples/examples in Mobius repo in debug mode, CSharpWorkerPath is required. Running DataFrame examples that do not use UDF do not require CSharpWorkerPath value to be specified

magmasystems

@magmasystems

Thanks very much, @skaarthik . That did the trick. I did have the CSharpWorkerPath value specified, but the path was to another version of CSharpWorker.exe in the root directory.

EdStone

@EdStone

Hi all

vignesh1453

@vignesh1453

Hi, I am new to Spark. How can I submit the word count(Mobius driver) program in Azure HDInsight Spark?

Stefan Mršević

@Mrsevic

I am experiencing the similar problem as the @magmasystems. I have CsharpWorkPath set up as well as the port, I have tried with the version from my /bin/debug and the driver version. I am trying to run Pi program.

The exact error is:

{"JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )"}

I have also tried with the temp parameter but it does not do any good as far as I have seen. I have completely disabled my AV software and I have also disabled MS Defender.

Thanks

Loch Wansbrough

@lwansbrough

Is it possible to use Mobius to create a custom RDD? And if so, is there an example of that? I essentially need spark-redis but written in C# with Mobius.

Loch Wansbrough

@lwansbrough

I see now that Mobius is essentially proxying requests to Spark. So I believe I'd need to have Mobius build spark with spark-redis, then I'd have to write the proxying code. Correct?

Marcin

@MarWin93

Hi guys, I want to do some math using Mobius. I got the rdd, similar to the Pi example and I want to compute factorial of each value in the Map function. But it seems that I cannot include the for loop inside the map function. Is there any way to compute the factorial?

Kaarthik Sivashanmugam

@skaarthik

@lwansbrough - correct

Kaarthik Sivashanmugam

@skaarthik

@MarWin93 - try using Reduce() on your RDD