Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
  • Oct 26 14:16
    harsh1107 commented #1328
  • Oct 26 14:15
    harsh1107 commented #1328
  • Oct 26 14:03
    harsh1107 commented #1328
  • Oct 26 14:00
    harsh1107 commented #1328
  • Oct 23 08:20
    sushanth-hi commented #1328
  • Oct 23 08:06
    sushanth-hi commented #1328
  • Oct 23 07:08
    Nibooor closed #1268
  • Oct 23 06:58
    Nibooor labeled #1275
  • Oct 23 06:57
    Nibooor labeled #1277
  • Oct 23 06:54
    valan4ik labeled #1315
  • Oct 23 06:54
    valan4ik closed #1315
  • Oct 23 06:54
    valan4ik commented #1315
  • Oct 23 06:54
    valan4ik labeled #1269
  • Oct 23 06:53
    Nibooor closed #1289
  • Oct 23 06:53
    Nibooor commented #1289
  • Oct 23 06:52
    Nibooor labeled #1289
  • Oct 23 06:52
    Nibooor labeled #1289
  • Oct 23 06:50
    Nibooor labeled #1290
  • Oct 23 06:45
    Nibooor closed #1303
  • Oct 23 06:44
    Nibooor labeled #1303
Behroz Sikander

well not really. As soon as the context is created the executors are spawned and they allocate resources.

Your only option is to use Dynamic Resource Allocation from Spark and it should work out of the box with jobserver (I hope).

Hi @bsikander @valan4ik , We need to set spray.can.server.keystorePW="<keystore password>" to enabled SSL on the spark job server. It is not secure as the env.conf will have the string form of the secure key. Do we have better approach to pass the keystore password? I am using Spark jobserver 0.9.
I submitted a Spark job on EMR cluster and specified 3 executors with 5 GB memory for each. But it is in accepted state for last 15-20 mins and when I checked available resources then I found that core nodes do not have enough memory available but task nodes have more than 50 GB memory available but still my job is in accepted state
So is it necessary to have required memory available on core nodes to start the spark application instead having resource available on task nodes
Hi Can some one please help me in setting up ssl , I am using CDH 5.15.1 with Spark job server version 0.8.0 using spark 2.2 . I have configured environment.conf spray.can.server { keystore = /opt/cloudera/security/pki/server.jks and trustore as /opt/cloudera/security/pki/ca-certs.jks } with the respective passwords. All my cloudera services work fine with ssl/tls enabled with the same settings
can some tell me if there is something wrong which i am doing
@pgouda89 can you please let me know how you have set up ssl
@bsikander can you please help me if there is something else which we need to configure to enable bothe client and server auth
Has anyone enabled ssl both server and client with job-server and CDH ?
ERROR erver.HttpServerConnection [] [akka.tcp://JobServer@xxxxxxxxxx:42392/user/IO-HTTP/listener-0/8] - Aborting encrypted connection to /XXXXXXXXX:xxxx due to [SSLHandshakeException:null cert chain] -> [SSLHandshakeException:null cert chain] is the error i face
i am using combinedDao with hdfs+postgres
and could not make it work. Can some one please help me with the syntax

jobdao = spark.jobserver.io.CombinedDAO

combineddao {
  rootdir = "/tmp/combineddao"
  binarydao {
    class = spark.jobserver.io.HdfsBinaryDAO

dit = "hdfs path"

metadatadao {
class = spark.jobserver.io.MetaDataSqlDAO

sqldao {

  # Slick database driver, full classpath
  slick-driver = slick.driver.PostgresDriver

  # JDBC driver, full classpath
  jdbc-driver = org.postgresql.Driver

url =

password =
flyway.locations="db/postgresql/migration ---> in the documentation it is given as db/combineddao/postgresql/migration
i see this error from spark job server Caused by: org.postgresql.util.PSQLException: ERROR: relation "BINARIES_CONTENTS" does not exist
and is unable to start
ERROR internal.command.DbMigrate [] [] - Migration of schema "public" to version 0.7.6 failed! Changes successfully rolled back.

Caused by: org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:

Migration V0_7_6__add_bin_hash_column.sql failed

SQL State : 42P01
Error Code : 0
Message : ERROR: relation "BINARIES_CONTENTS" does not exist
Location : db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql (/data01/spark/spark-jobserver/job-server/file:/data01/spark/spark-jobserver/job-server/spark-job-server.jar!/db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql)
Line : 4

@bsikander @noorul can you guys please help me with this
0.8.0 sjs version using with spark 2.2 on cdh 5.15.1

@Narasimhaporeddy Hi, I think you may have some documentation/jobserver versions mismatch:
if you use jobserver 0.8.0, please check documentation for this version: https://github.com/spark-jobserver/spark-jobserver/tree/0.8.0

db/combineddao/postgresql/migration was introduced only recently and is not part of 0.8.0 release
I think hdfs+postgres DAO would work only if you use master branch
You may use version 0.10.0 and use hdfs+h2 DAO (https://github.com/spark-jobserver/spark-jobserver/tree/e3c3d3ce9ba81b63608130d3904161c8246fe064)

@valan4ik thanks for pointing that out to me. Indeed i was referring to latest documentation. One quick Question does the latest version support spark 2.2.0 ?
@Narasimhaporeddy Some dependencies and queries were updated, I am not sure if it is fully backward compatible. You could give it a try by setting SPARK_VERSION variable in your configuration file :)
I think there is no other way to try new DAOs
aww makes sense. @valan4ik thanks for the quick help :)
Hey did any one enable kerberos in sjs 0.8.0 , can you please let me know how you configured it
i did try to set up use-as-proxy-user=on in shiro.ini and try to export the keytab as mentioned in some of the knime documentation but it did not work for me.
@valan4ik @bsikander i did not see any specific doc on enabling kerberos in specific for spark job server can you please help me with this
Hello All,Can anybody help me how to use the spark-jobserver in windows and possible pls share any link.
Evan Chan
@sudhakarkunchala what issue do you have on Windows? It is JVM based, in theory should work just fine....
Lomesh Agrawal
Does SJS compile with Scala 2.12?
Krishna sanjay
how to build spark-job-server without using sbt
I am using Scala 2.12. When I built the program, I couldn't download io.spray jars. Doesn't it mean Scala 2.12 isn't supported yet?
Krishna sanjay
spark job server https enable?
@Lomesh941 have u compiled and got the jar?
I got an error msg, when calling a job: "result": "classPath not found"
}),404,Not Found,Vector((Server,spray-can/1.3.4), (Access-Control-Allow-Origin,*), (Content-Length,59), (Date,Thu, 30 Jul 2020 02:34:10 GMT), (Content-Type,application/json; charset=UTF-8)),List())
Open an issue: spark-jobserver/spark-jobserver#1317
Could anyboy help?

Hi @valan4ik , Do we support utf-8 spark context name? for example: curl -i -d "" 'http://<host>:8090/contexts/Sparkconext漢字?num-cpu-cores=2&memory-per-node=512M&context-factory=spark.jobserver.context.SessionContextFactory' . I got followign error upon executing the above curl command: HTTP/1.1 400 Bad Request
Server: spray-can/1.3.4
Date: Fri, 31 Jul 2020 00:00:36 GMT
Content-Type: text/plain; charset=UTF-8
Connection: close
Content-Length: 65

Illegal request-target, unexpected character '₩' at position 23-bash-4.2$

Hey guys, quick question about the "max-jobs-per-context" setting. I'm running with context-per-jvm but it would be great if I could set different max-jobs limit for each context individually. Checking the usage of this setting in the source files, it only seems related to the "akka.threads" setting in the Spark Application, but I thought those was already deprecated and therefore useless (using Spark 2.3.2). Any ideas on how could I achieve this?
Hi .. Is Spark 3.0 supported in 0.10.0?
harshal patil

Hi all , I was running concurrency benchmark on spark-job-server using Jmeter, but I am not able to achieve high concurrency with increasing cores .

override def runJob(sparkSession: SparkSession, runtime: JobEnvironment, data: JobData): JobOutput = { Map("data" -> 1) }

I am not running any spark job here .
still I am not able to achieve more than 8 query per second -> Same results on (4 ,8 , 16) core aws ec2 machine

I have created 4 contexts and maintaining concurrency 5 per context

max-jobs-per-context = num of cores on machine

Can anyone tell me , what could be going wrong here ?