Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 26 14:16
    harsh1107 commented #1328
  • Oct 26 14:15
    harsh1107 commented #1328
  • Oct 26 14:03
    harsh1107 commented #1328
  • Oct 26 14:00
    harsh1107 commented #1328
  • Oct 23 08:20
    sushanth-hi commented #1328
  • Oct 23 08:06
    sushanth-hi commented #1328
  • Oct 23 07:08
    Nibooor closed #1268
  • Oct 23 06:58
    Nibooor labeled #1275
  • Oct 23 06:57
    Nibooor labeled #1277
  • Oct 23 06:54
    valan4ik labeled #1315
  • Oct 23 06:54
    valan4ik closed #1315
  • Oct 23 06:54
    valan4ik commented #1315
  • Oct 23 06:54
    valan4ik labeled #1269
  • Oct 23 06:53
    Nibooor closed #1289
  • Oct 23 06:53
    Nibooor commented #1289
  • Oct 23 06:52
    Nibooor labeled #1289
  • Oct 23 06:52
    Nibooor labeled #1289
  • Oct 23 06:50
    Nibooor labeled #1290
  • Oct 23 06:45
    Nibooor closed #1303
  • Oct 23 06:44
    Nibooor labeled #1303
Narasimhaporeddy
@Narasimhaporeddy
}
metadatadao {
class = spark.jobserver.io.MetaDataSqlDAO
}

sqldao {

  # Slick database driver, full classpath
  slick-driver = slick.driver.PostgresDriver

  # JDBC driver, full classpath
  jdbc-driver = org.postgresql.Driver

url =

user=
password =
flyway.locations="db/postgresql/migration ---> in the documentation it is given as db/combineddao/postgresql/migration
i see this error from spark job server Caused by: org.postgresql.util.PSQLException: ERROR: relation "BINARIES_CONTENTS" does not exist
and is unable to start
ERROR internal.command.DbMigrate [] [] - Migration of schema "public" to version 0.7.6 failed! Changes successfully rolled back.

Caused by: org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:

Migration V0_7_6__add_bin_hash_column.sql failed

SQL State : 42P01
Error Code : 0
Message : ERROR: relation "BINARIES_CONTENTS" does not exist
Location : db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql (/data01/spark/spark-jobserver/job-server/file:/data01/spark/spark-jobserver/job-server/spark-job-server.jar!/db/postgresql/migration/V0_7_6/V0_7_6add_bin_hash_column.sql)
Line : 4

Statement : ALTER TABLE "BINARIES_CONTENTS" ADD COLUMN "BIN_HASH" BYTEA
@bsikander @noorul can you guys please help me with this
Narasimhaporeddy
@Narasimhaporeddy
0.8.0 sjs version using with spark 2.2 on cdh 5.15.1
Valentina
@valan4ik

@Narasimhaporeddy Hi, I think you may have some documentation/jobserver versions mismatch:
if you use jobserver 0.8.0, please check documentation for this version: https://github.com/spark-jobserver/spark-jobserver/tree/0.8.0

db/combineddao/postgresql/migration was introduced only recently and is not part of 0.8.0 release
I think hdfs+postgres DAO would work only if you use master branch
You may use version 0.10.0 and use hdfs+h2 DAO (https://github.com/spark-jobserver/spark-jobserver/tree/e3c3d3ce9ba81b63608130d3904161c8246fe064)

Narasimhaporeddy
@Narasimhaporeddy
@valan4ik thanks for pointing that out to me. Indeed i was referring to latest documentation. One quick Question does the latest version support spark 2.2.0 ?
Valentina
@valan4ik
@Narasimhaporeddy Some dependencies and queries were updated, I am not sure if it is fully backward compatible. You could give it a try by setting SPARK_VERSION variable in your configuration file :)
I think there is no other way to try new DAOs
Narasimhaporeddy
@Narasimhaporeddy
aww makes sense. @valan4ik thanks for the quick help :)
Narasimhaporeddy
@Narasimhaporeddy
Hey did any one enable kerberos in sjs 0.8.0 , can you please let me know how you configured it
i did try to set up use-as-proxy-user=on in shiro.ini and try to export the keytab as mentioned in some of the knime documentation but it did not work for me.
@valan4ik @bsikander i did not see any specific doc on enabling kerberos in specific for spark job server can you please help me with this
sudhakarkunchala
@sudhakarkunchala
Hello All,Can anybody help me how to use the spark-jobserver in windows and possible pls share any link.
Evan Chan
@velvia
@sudhakarkunchala what issue do you have on Windows? It is JVM based, in theory should work just fine....
Lomesh Agrawal
@Lomesh941
Does SJS compile with Scala 2.12?
Krishna sanjay
@cris7M
how to build spark-job-server without using sbt
calvinzhan
@calvinzhan
I am using Scala 2.12. When I built the program, I couldn't download io.spray jars. Doesn't it mean Scala 2.12 isn't supported yet?
Krishna sanjay
@cris7M
spark job server https enable?
@Lomesh941 have u compiled and got the jar?
calvinzhan
@calvinzhan
hi
I got an error msg, when calling a job: "result": "classPath not found"
}),404,Not Found,Vector((Server,spray-can/1.3.4), (Access-Control-Allow-Origin,*), (Content-Length,59), (Date,Thu, 30 Jul 2020 02:34:10 GMT), (Content-Type,application/json; charset=UTF-8)),List())
Open an issue: spark-jobserver/spark-jobserver#1317
Could anyboy help?
pgouda89
@pgouda89

Hi @valan4ik , Do we support utf-8 spark context name? for example: curl -i -d "" 'http://<host>:8090/contexts/Sparkconext漢字?num-cpu-cores=2&memory-per-node=512M&context-factory=spark.jobserver.context.SessionContextFactory' . I got followign error upon executing the above curl command: HTTP/1.1 400 Bad Request
Server: spray-can/1.3.4
Date: Fri, 31 Jul 2020 00:00:36 GMT
Content-Type: text/plain; charset=UTF-8
Connection: close
Content-Length: 65

Illegal request-target, unexpected character '₩' at position 23-bash-4.2$

kz3r
@kz3r
Hey guys, quick question about the "max-jobs-per-context" setting. I'm running with context-per-jvm but it would be great if I could set different max-jobs limit for each context individually. Checking the usage of this setting in the source files, it only seems related to the "akka.threads" setting in the Spark Application, but I thought those was already deprecated and therefore useless (using Spark 2.3.2). Any ideas on how could I achieve this?
pgouda89
@pgouda89
Hi .. Is Spark 3.0 supported in 0.10.0?
harshal patil
@harshal1107_twitter

Hi all , I was running concurrency benchmark on spark-job-server using Jmeter, but I am not able to achieve high concurrency with increasing cores .

override def runJob(sparkSession: SparkSession, runtime: JobEnvironment, data: JobData): JobOutput = { Map("data" -> 1) }

I am not running any spark job here .
still I am not able to achieve more than 8 query per second -> Same results on (4 ,8 , 16) core aws ec2 machine

I have created 4 contexts and maintaining concurrency 5 per context

max-jobs-per-context = num of cores on machine

Can anyone tell me , what could be going wrong here ?