Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 23 09:15

    vglagoleva on master

    feat(jobserver): Add result per… feat(jobserver): Replace result… fix(webapi): Jobs started in sy… and 18 more (compare)

  • Jul 23 09:15
    vglagoleva closed #1375
  • Jul 22 08:28
    codecov-commenter commented #1375
  • Jul 22 07:56
    Nibooor synchronize #1375
  • Jul 21 10:55
    Nibooor synchronize #1375
  • Jul 19 13:51
    Nibooor synchronize #1375
  • Jul 19 13:43

    Nibooor on docker_sbt

    (compare)

  • Jul 19 13:43

    Nibooor on master

    Fix installation of SBT in Trav… (compare)

  • Jul 19 13:43
    Nibooor closed #1377
  • Jul 15 15:44
    vglagoleva review_requested #1377
  • Jul 15 15:36
    vglagoleva synchronize #1377
  • Jul 15 15:36

    vglagoleva on docker_sbt

    Fix installation of SBT in Trav… (compare)

  • Jul 15 14:57
    codecov-commenter commented #1377
  • Jul 15 14:43
    codecov-commenter commented #1377
  • Jul 15 14:25
    vglagoleva synchronize #1377
  • Jul 15 14:25

    vglagoleva on docker_sbt

    Fix installation of SBT in Trav… (compare)

  • Jul 15 14:05
    vglagoleva edited #1377
  • Jul 15 14:04
    vglagoleva synchronize #1377
  • Jul 15 14:04

    vglagoleva on docker_sbt

    Fix installation of SBT in Trav… (compare)

  • Jul 15 13:58
    zeitos edited #1377
Narasimhaporeddy
@Narasimhaporeddy
@bsikander @noorul can you guys please help me with this
0.8.0 sjs version using with spark 2.2 on cdh 5.15.1
Valentina Glagoleva
@valan4ik

@Narasimhaporeddy Hi, I think you may have some documentation/jobserver versions mismatch:
if you use jobserver 0.8.0, please check documentation for this version: https://github.com/spark-jobserver/spark-jobserver/tree/0.8.0

db/combineddao/postgresql/migration was introduced only recently and is not part of 0.8.0 release
I think hdfs+postgres DAO would work only if you use master branch
You may use version 0.10.0 and use hdfs+h2 DAO (https://github.com/spark-jobserver/spark-jobserver/tree/e3c3d3ce9ba81b63608130d3904161c8246fe064)

Narasimhaporeddy
@Narasimhaporeddy
@valan4ik thanks for pointing that out to me. Indeed i was referring to latest documentation. One quick Question does the latest version support spark 2.2.0 ?
Valentina Glagoleva
@valan4ik
@Narasimhaporeddy Some dependencies and queries were updated, I am not sure if it is fully backward compatible. You could give it a try by setting SPARK_VERSION variable in your configuration file :)
I think there is no other way to try new DAOs
Narasimhaporeddy
@Narasimhaporeddy
aww makes sense. @valan4ik thanks for the quick help :)
Narasimhaporeddy
@Narasimhaporeddy
Hey did any one enable kerberos in sjs 0.8.0 , can you please let me know how you configured it
i did try to set up use-as-proxy-user=on in shiro.ini and try to export the keytab as mentioned in some of the knime documentation but it did not work for me.
@valan4ik @bsikander i did not see any specific doc on enabling kerberos in specific for spark job server can you please help me with this
sudhakarkunchala
@sudhakarkunchala
Hello All,Can anybody help me how to use the spark-jobserver in windows and possible pls share any link.
Evan Chan
@velvia
@sudhakarkunchala what issue do you have on Windows? It is JVM based, in theory should work just fine....
Lomesh Agrawal
@Lomesh941
Does SJS compile with Scala 2.12?
Krishna sanjay
@cris7M
how to build spark-job-server without using sbt
calvinzhan
@calvinzhan
I am using Scala 2.12. When I built the program, I couldn't download io.spray jars. Doesn't it mean Scala 2.12 isn't supported yet?
Krishna sanjay
@cris7M
spark job server https enable?
@Lomesh941 have u compiled and got the jar?
calvinzhan
@calvinzhan
hi
I got an error msg, when calling a job: "result": "classPath not found"
}),404,Not Found,Vector((Server,spray-can/1.3.4), (Access-Control-Allow-Origin,*), (Content-Length,59), (Date,Thu, 30 Jul 2020 02:34:10 GMT), (Content-Type,application/json; charset=UTF-8)),List())
Open an issue: spark-jobserver/spark-jobserver#1317
Could anyboy help?
pgouda89
@pgouda89

Hi @valan4ik , Do we support utf-8 spark context name? for example: curl -i -d "" 'http://<host>:8090/contexts/Sparkconext漢字?num-cpu-cores=2&memory-per-node=512M&context-factory=spark.jobserver.context.SessionContextFactory' . I got followign error upon executing the above curl command: HTTP/1.1 400 Bad Request
Server: spray-can/1.3.4
Date: Fri, 31 Jul 2020 00:00:36 GMT
Content-Type: text/plain; charset=UTF-8
Connection: close
Content-Length: 65

Illegal request-target, unexpected character '₩' at position 23-bash-4.2$

kz3r
@kz3r
Hey guys, quick question about the "max-jobs-per-context" setting. I'm running with context-per-jvm but it would be great if I could set different max-jobs limit for each context individually. Checking the usage of this setting in the source files, it only seems related to the "akka.threads" setting in the Spark Application, but I thought those was already deprecated and therefore useless (using Spark 2.3.2). Any ideas on how could I achieve this?
pgouda89
@pgouda89
Hi .. Is Spark 3.0 supported in 0.10.0?
harshal patil
@harshal1107_twitter

Hi all , I was running concurrency benchmark on spark-job-server using Jmeter, but I am not able to achieve high concurrency with increasing cores .

override def runJob(sparkSession: SparkSession, runtime: JobEnvironment, data: JobData): JobOutput = { Map("data" -> 1) }

I am not running any spark job here .
still I am not able to achieve more than 8 query per second -> Same results on (4 ,8 , 16) core aws ec2 machine

I have created 4 contexts and maintaining concurrency 5 per context

max-jobs-per-context = num of cores on machine

Can anyone tell me , what could be going wrong here ?

sj123050037
@sj123050037
Hello, I want to know if SparkJobServer can manage the jobs/contexts on a Kubernetes cluster?
sj123050037
@sj123050037
hi @bsikander, I am using Spark jobserver in Yarn cluster mode. I have few questions, can you please answer the following for Yarn cluster mode:-
  1. If I am running in Yarn cluster mode, what does num-cpu-cores control? I know that it gets translated to spark.cores.max but this property is irrelevant for the Yarn-based Spark context.
  2. If I set the max-jobs-per-context to a very high value, then persistent Spark context should be able to support any concurrency (limited by the resources on the yarn cluster)
Behroz Sikander
@bsikander
@pgouda89 you can check the compatibility table (https://github.com/spark-jobserver/spark-jobserver#version-information). It is currently not supported but the support is easy to add. Please feel free to try it out, if you need help, we can guide you.
Behroz Sikander
@bsikander
@harshal1107_twitter sorry for a late reply. I don't get your problem exactly. You have launched 4 contexts and then you submit jobs to them. At max, you are able to launch only 8 jobs across 4 contexts?
Behroz Sikander
@bsikander
@sj123050037
1- This property should just get ignored. Jobserver sets this property in SparkConf and yarn will just not use it.
2- Interestingly, i think that this property should be removed since this property is translated to "spark.akka.threads" and this property has been removed from Spark for quite some time. This property will also not have any effect
sj123050037
@sj123050037
Thanks for clearing this @bsikander! Maybe we can update the documentation to indicate this specifically?
Ismail
@IsmailEL_gitlab
Hi
image.png
guys I hve this issue while connecting to Mariadb
Any ideas ? tks in advance
Locally it works but while executing it on hdfs it cries
Behroz Sikander
@bsikander
@sj123050037 feel free to contribute :)
Behroz Sikander
@bsikander
@IsmailEL_gitlab what is the error? most of the log lines in the screenshot are showing driver's normal shutdown
Ismail
@IsmailEL_gitlab
Hi @bsikander thanks for reply, I can run the jar file only if I add the path of mysql connector driver in my cmd
But when I don't give the whole path (even if it's already declared in sbt file) it does not run and gives me that error.
Behroz Sikander
@bsikander
@IsmailEL_gitlab can you give some more details on what you are trying to do? Are you trying to run jobserver with mariadb or trying to use mariadb in your driver/executor code?
Also please open a bug in jobserver so it can be tracked better
Ismail
@IsmailEL_gitlab
Hi @bsikander , Yes here some details : I'm running a project using Mongodb, hadoop cluster and Mariadb, this last one is my datawarehouse, I do a short treatment then I push data in dwh, but when I don't specify the whole path of mariadb's connector, it crashs and doesn't find it even if I already added the lib in sbt file.
Idk if i was clear enough this time..
Don't hesitate to ask for more if needed
Ismail
@IsmailEL_gitlab
Cannot connect to mariadb without giving the whole path in execution command line #1334 done to track !
sj123050037
@sj123050037
Hello guys, Are we planning to support to Spark-3.0.0 anytime soon?
Valentina Glagoleva
@valan4ik
Hi @sj123050037, there is a feature request for it: spark-jobserver/spark-jobserver#1269
Contributions are welcome :)
Behroz Sikander
@bsikander
@IsmailEL_gitlab @valan4ik replied you already in the ticket.
@sj123050037 adding to @valan4ik's answer, Scala 2.12 support has already been added to jobserver. Contributing 3.0.0 should be really straight forward.
sj123050037
@sj123050037
Thanks, @bsikander! I have upgraded Spark to 3.0.1 in my setup. It works with a few deprecations. I am able to run in local mode and tests are passing too.
Behroz Sikander
@bsikander
great, looking forward to your patch.
pgouda89
@pgouda89
Hi All, anyone here using SJS 0.10 or up in yarn cluster mode? I am seeing spark-jobserver/spark-jobserver#1371 issue after I have upgraded mine from SJS 0.9 to 0.11.