To reproduce your exception, I submitted a Pi example(https://github.com/Microsoft/SparkCLR/tree/master/examples/pi) to yarn cluster with following cmd:
scripts\sparkclr-submit.cmd --verbose --master yarn-cluster --exe Pi.exe ..\Pi.zip
Unfortunately I can't reproduce it. Please double check the environment variables including SPARK_HOME, HADOOP_HOME and make sure they are set properly, and the configuration items are also consistent with the yarn cluster your application will be submitted to.
SparkContext = new SparkContext("yarn-cluster", "PiExample")
to instantiate the sparkcontext.
===== Building F:\SparkCLR\csharp\SparkCLR.sln =====
Restore NuGet packages ===================
MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86
)\MSBuild\14.0\bin'.
All packages listed in packages.config are already installed.
Build Debug ==============================
Microsoft (R) Build Engine version 4.6.1055.0
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.
F:\SparkCLR\csharp\SparkCLR.sln.metaproj : error MSB4126: The specified solutio
n configuration "Debug|x64" is invalid. Please specify a valid solution configu
ration using the Configuration and Platform properties (e.g. MSBuild.exe Soluti
on.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties
blank to use the default solution configuration. [F:\SparkCLR\csharp\SparkCLR.s
ln]
===== Build FAILED for F:\SparkCLR\csharp\SparkCLR.sln -- Debug with error 1 - C
ANNOT CONTINUE =====
Build SparkCLR C# components failed, stop building.
Hi Kaarthik. I just cloned your Git repo. I am not really a Spark developer, so my question might be fairly elemental. I got everything building, and the unit tests run just fine. What I want to do is debug the Pi sample from within Visual Studio 2015. I first opened up a Visual Studio Developer's Command Prompt and as per your debug instructions, ran the command "sparkclr-submit.cmd debug". I got the message "[CSharpRunner.main] Backend running debug mode. Press enter to exit" .... so far, so good. Then, I opened the SparkCLR.sln solution in Visual Studio, set the startup program to the "Samples" project, and in the project properties, I set the arguments to "--torun pi*". I then started the debug session. After the 300,000 integer array was initialized, I got these errors:
JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=13], )
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.net.SocketException: Connection reset by peer: socket write error
Do you have any idea? I am sure that I need to set some jar file path or class path somewhere.