Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 28 11:09
    valan4ik closed #325
  • Nov 28 11:09
    valan4ik commented #325
  • Nov 27 20:02
    zeitos edited #1331
  • Nov 27 20:02
    valan4ik review_requested #1331
  • Nov 27 20:02
    valan4ik review_requested #1331
  • Nov 27 20:02
    valan4ik review_requested #1331
  • Nov 27 20:02
    valan4ik opened #1331
  • Nov 27 20:02
    valan4ik review_requested #1331
  • Nov 27 20:00

    valan4ik on 0101_notes

    notes(jobserver): Add release n… (compare)

  • Nov 27 19:54
    valan4ik closed #1306
  • Nov 27 19:54
    valan4ik commented #1306
  • Nov 27 19:32

    valan4ik on v0.10.1

    (compare)

  • Nov 27 19:32

    valan4ik on jobserver-0.10.x

    Setting version to 0.10.1 Setting version to 0.10.2-SNAPS… (compare)

  • Nov 27 12:14
    valan4ik review_requested #1329
  • Nov 27 12:13

    valan4ik on jobserver-0.10.x

    fix: Fail postgres migration on… (compare)

  • Nov 27 12:13
    valan4ik closed #1329
  • Nov 27 10:24
    Ennosigaeon commented #1329
  • Nov 27 10:23
    Ennosigaeon synchronize #1329
  • Nov 27 06:30
    valan4ik commented #1329
  • Nov 27 06:30
    valan4ik commented #1329
Narasimhaporeddy
@Narasimhaporeddy

Caused by: org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:

Migration V0_7_6__add_bin_hash_column.sql failed

SQL State : 42P01
Error Code : 0
Message : ERROR: relation "BINARIES_CONTENTS" does not exist
Location : db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql (/data01/spark/spark-jobserver/job-server/file:/data01/spark/spark-jobserver/job-server/spark-job-server.jar!/db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql)
Line : 4

Statement : ALTER TABLE "BINARIES_CONTENTS" ADD COLUMN "BIN_HASH" BYTEA
@bsikander @noorul can you guys please help me with this
0.8.0 sjs version using with spark 2.2 on cdh 5.15.1
Valentina
@valan4ik

@Narasimhaporeddy Hi, I think you may have some documentation/jobserver versions mismatch:
if you use jobserver 0.8.0, please check documentation for this version: https://github.com/spark-jobserver/spark-jobserver/tree/0.8.0

db/combineddao/postgresql/migration was introduced only recently and is not part of 0.8.0 release
I think hdfs+postgres DAO would work only if you use master branch
You may use version 0.10.0 and use hdfs+h2 DAO (https://github.com/spark-jobserver/spark-jobserver/tree/e3c3d3ce9ba81b63608130d3904161c8246fe064)

Narasimhaporeddy
@Narasimhaporeddy
@valan4ik thanks for pointing that out to me. Indeed i was referring to latest documentation. One quick Question does the latest version support spark 2.2.0 ?
Valentina
@valan4ik
@Narasimhaporeddy Some dependencies and queries were updated, I am not sure if it is fully backward compatible. You could give it a try by setting SPARK_VERSION variable in your configuration file :)
I think there is no other way to try new DAOs
Narasimhaporeddy
@Narasimhaporeddy
aww makes sense. @valan4ik thanks for the quick help :)
Narasimhaporeddy
@Narasimhaporeddy
Hey did any one enable kerberos in sjs 0.8.0 , can you please let me know how you configured it
i did try to set up use-as-proxy-user=on in shiro.ini and try to export the keytab as mentioned in some of the knime documentation but it did not work for me.
@valan4ik @bsikander i did not see any specific doc on enabling kerberos in specific for spark job server can you please help me with this
sudhakarkunchala
@sudhakarkunchala
Hello All,Can anybody help me how to use the spark-jobserver in windows and possible pls share any link.
Evan Chan
@velvia
@sudhakarkunchala what issue do you have on Windows? It is JVM based, in theory should work just fine....
Lomesh Agrawal
@Lomesh941
Does SJS compile with Scala 2.12?
Krishna sanjay
@cris7M
how to build spark-job-server without using sbt
calvinzhan
@calvinzhan
I am using Scala 2.12. When I built the program, I couldn't download io.spray jars. Doesn't it mean Scala 2.12 isn't supported yet?
Krishna sanjay
@cris7M
spark job server https enable?
@Lomesh941 have u compiled and got the jar?
calvinzhan
@calvinzhan
hi
I got an error msg, when calling a job: "result": "classPath not found"
}),404,Not Found,Vector((Server,spray-can/1.3.4), (Access-Control-Allow-Origin,*), (Content-Length,59), (Date,Thu, 30 Jul 2020 02:34:10 GMT), (Content-Type,application/json; charset=UTF-8)),List())
Open an issue: spark-jobserver/spark-jobserver#1317
Could anyboy help?
pgouda89
@pgouda89

Hi @valan4ik , Do we support utf-8 spark context name? for example: curl -i -d "" 'http://<host>:8090/contexts/Sparkconext漢字?num-cpu-cores=2&memory-per-node=512M&context-factory=spark.jobserver.context.SessionContextFactory' . I got followign error upon executing the above curl command: HTTP/1.1 400 Bad Request
Server: spray-can/1.3.4
Date: Fri, 31 Jul 2020 00:00:36 GMT
Content-Type: text/plain; charset=UTF-8
Connection: close
Content-Length: 65

Illegal request-target, unexpected character '₩' at position 23-bash-4.2$

kz3r
@kz3r
Hey guys, quick question about the "max-jobs-per-context" setting. I'm running with context-per-jvm but it would be great if I could set different max-jobs limit for each context individually. Checking the usage of this setting in the source files, it only seems related to the "akka.threads" setting in the Spark Application, but I thought those was already deprecated and therefore useless (using Spark 2.3.2). Any ideas on how could I achieve this?
pgouda89
@pgouda89
Hi .. Is Spark 3.0 supported in 0.10.0?
harshal patil
@harshal1107_twitter

Hi all , I was running concurrency benchmark on spark-job-server using Jmeter, but I am not able to achieve high concurrency with increasing cores .

override def runJob(sparkSession: SparkSession, runtime: JobEnvironment, data: JobData): JobOutput = { Map("data" -> 1) }

I am not running any spark job here .
still I am not able to achieve more than 8 query per second -> Same results on (4 ,8 , 16) core aws ec2 machine

I have created 4 contexts and maintaining concurrency 5 per context

max-jobs-per-context = num of cores on machine

Can anyone tell me , what could be going wrong here ?

sj123050037
@sj123050037
Hello, I want to know if SparkJobServer can manage the jobs/contexts on a Kubernetes cluster?
sj123050037
@sj123050037
hi @bsikander, I am using Spark jobserver in Yarn cluster mode. I have few questions, can you please answer the following for Yarn cluster mode:-
  1. If I am running in Yarn cluster mode, what does num-cpu-cores control? I know that it gets translated to spark.cores.max but this property is irrelevant for the Yarn-based Spark context.
  2. If I set the max-jobs-per-context to a very high value, then persistent Spark context should be able to support any concurrency (limited by the resources on the yarn cluster)
Behroz Sikander
@bsikander
@pgouda89 you can check the compatibility table (https://github.com/spark-jobserver/spark-jobserver#version-information). It is currently not supported but the support is easy to add. Please feel free to try it out, if you need help, we can guide you.
Behroz Sikander
@bsikander
@harshal1107_twitter sorry for a late reply. I don't get your problem exactly. You have launched 4 contexts and then you submit jobs to them. At max, you are able to launch only 8 jobs across 4 contexts?
Behroz Sikander
@bsikander
@sj123050037
1- This property should just get ignored. Jobserver sets this property in SparkConf and yarn will just not use it.
2- Interestingly, i think that this property should be removed since this property is translated to "spark.akka.threads" and this property has been removed from Spark for quite some time. This property will also not have any effect
sj123050037
@sj123050037
Thanks for clearing this @bsikander! Maybe we can update the documentation to indicate this specifically?