Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Avi Avni
    @AviAvni
    Hii i'm new to spark and using it with c# is graet what i need to do to debug the samples i add to the app setting CSharpBackendPortNumber what i don't understand what i need to run spark-shell/spark-submit?
    Avi Avni
    @AviAvni
    The wiki help me i can debug
    Kaarthik Sivashanmugam
    @skaarthik
    we had instructions for debugging somewhere in the repo...let me get the link and post it here
    Avi Avni
    @AviAvni
    I can debug the pi sample but i have problen with the data frame sample i have error import sqlcontext
    Kaarthik Sivashanmugam
    @skaarthik
    what is the error? Is it on missing spark-csv*.jar file?
    Avi Avni
    @AviAvni
    System.Exception: JVM method execution failed: Nonstatic method jdbc failed for class 7 when called with 3 parameters ([Index=1, Type=String, Value=jdbc:sqlserver://localhost:1433;databaseName=Temp;;integratedSecurity=true;], [Index=2, Type=String, Value=xyz], [Index=3, Type=JvmObjectReference, Value=8], )
    at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
    at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object[] parameters)
    at Microsoft.Spark.CSharp.Proxy.Ipc.DataFrameReaderIpcProxy.Jdbc(String url, String table, Dictionary2 properties) at Microsoft.Spark.CSharp.Sql.DataFrameReader.Jdbc(String url, String table, Dictionary2 properties)
    at Microsoft.Spark.CSharp.Examples.JdbcDataFrameExample.Main(String[] args) in C:\Users\Avi\Downloads\spark-clr_2.10-1.5.200-preview-1\examples\JdbcDataFrame\Program.cs:line 26
    Kaarthik Sivashanmugam
    @skaarthik
    this is probably because JDBC jar file is not in the classpath
    Avi Avni
    @AviAvni
    jdbc.jar file not exist in my computer where to get it
    Kaarthik Sivashanmugam
    @skaarthik
    not the ideal way but a quick workaround is to set "SPARKCSV_JARS" environment variable pointing to the jar file in command prompt before launching "sparkclr-submit.cmd debug"
    use your favorite search engine to look for sql server jdbc and download jdbc jar file your version of sql server
    Avi Avni
    @AviAvni
    thanks i'll try
    there is connection to graphx?
    Kaarthik Sivashanmugam
    @skaarthik
    I do not know if GraphX supports jdbc (I guess that's what you are asking)
    vaskokj
    @vaskokj
    when calling "var sparkContext = new SparkContext("yarn-cluster", "ExampleAppName"); I get this exception. Additional information: JVM method execution failed: Constructor failed for class org.apache.spark.SparkContext when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=8], ). Is it not possible to call the API to run on a remote Spark cluster?
    Tao Wang
    @tawan0109

    To reproduce your exception, I submitted a Pi example(https://github.com/Microsoft/SparkCLR/tree/master/examples/pi) to yarn cluster with following cmd:
    scripts\sparkclr-submit.cmd --verbose --master yarn-cluster --exe Pi.exe ..\Pi.zip

    Unfortunately I can't reproduce it. Please double check the environment variables including SPARK_HOME, HADOOP_HOME and make sure they are set properly, and the configuration items are also consistent with the yarn cluster your application will be submitted to.

    For the Pi example I submitted, I changed it a little bit and use SparkContext = new SparkContext("yarn-cluster", "PiExample") to instantiate the sparkcontext.
    Harsha Nihanth
    @spiharsh
    Hi, I am trying to build the solution as per the instructions, after running Build.cmd i am facing with the MSBuild error MSB4126 stating that the configuration Debug|x64 is not valid
    I looked into the .sln file and see that it no where mentioned as x64. Can someone help me out in understanding the issue I am facing here. Thanks.
    Kaarthik Sivashanmugam
    @skaarthik
    @spiharsh - where do you see the MSBuild error? In Visual Studio or in the command line output from Build.cmd? I am not able to reproduce the error you got.
    Harsha Nihanth
    @spiharsh
    From command line build.cmd
    When I build it using Visual Studio I don't see any errors.
    This is the log of the error
    ===== Building F:\SparkCLR\csharp\SparkCLR.sln ===== Restore NuGet packages =================== MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86 )\MSBuild\14.0\bin'. All packages listed in packages.config are already installed. Build Debug ============================== Microsoft (R) Build Engine version 4.6.1055.0 [Microsoft .NET Framework, version 4.0.30319.42000] Copyright (C) Microsoft Corporation. All rights reserved. F:\SparkCLR\csharp\SparkCLR.sln.metaproj : error MSB4126: The specified solutio n configuration "Debug|x64" is invalid. Please specify a valid solution configu ration using the Configuration and Platform properties (e.g. MSBuild.exe Soluti on.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties blank to use the default solution configuration. [F:\SparkCLR\csharp\SparkCLR.s ln] ===== Build FAILED for F:\SparkCLR\csharp\SparkCLR.sln -- Debug with error 1 - C ANNOT CONTINUE ===== Build SparkCLR C# components failed, stop building.
    Kaarthik Sivashanmugam
    @skaarthik
    I guess I could not repro the error because I have MSBuild 12.0 in my machine and looks like you are using 14.0...if you have 12.0, try using that to version of MSBuild to build SparkCLR.
    guwang
    @jayjaywg
    @spiharsh can you try MSBuild "D:\SparkCLR\csharp\SparkCLR.sln" /property:Configuration="Debug" /property:Platform="Any CPU"?
    Daniel Li
    @danielli90
    Tried both VS 2015 RTM and VS 2015 Upgrade 1, which has the latest MS build engine version (4.6.1055.0). No repro. @spiharsh, are you running the build.cmd under a 32-bit Windows?
    Eduardo Gonzalez
    @wmeddie
    I heard rumors of a release?
    Harsha Nihanth
    @spiharsh
    @jayjaywg using msbuild inside \chsarp with /p:Configuration="Debug" /p:Platform="Any CPU" worked
    @danielli90 I am under 64bit windows
    Harsha Nihanth
    @spiharsh
    When I include the argument /p:Platform="Any CPU" in build.cmd of sparkclr\charp and sparkclr\examples, the build works fine .