Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 15 09:21
    dos65 closed #549
  • Aug 15 09:21
    dos65 opened #549
  • Jul 30 16:06
    dos65 closed #450
  • Jul 30 16:05
    dos65 closed #339
  • Jul 30 16:05
    dos65 closed #509
  • Jul 30 16:03
    dos65 closed #543
  • Jul 30 16:02
    dos65 labeled #548
  • Jul 30 16:02
    dos65 opened #548
  • Jul 24 13:34
    dos65 closed #547
  • Jul 23 05:37
    dos65 synchronize #547
  • Jul 23 05:21
    dos65 opened #547
  • Jul 03 01:36
    otto-dev commented #546
  • Jul 02 11:55
    dos65 commented #546
  • Jul 02 03:51
    otto-dev reopened #546
  • Jul 02 03:50
    otto-dev closed #546
  • Jul 02 02:19
    otto-dev commented #546
  • Jul 02 02:19
    otto-dev edited #546
  • Jul 02 01:55
    otto-dev edited #546
  • Jul 02 01:53
    otto-dev opened #546
  • Jun 20 18:00
    danielhanbitlee commented #545
Ali Emirhan Kurt
@AEKurt
when i create function it ask me the path of my application jar
as you know we gives local file system
i think that is a problem when i run my application as yarn cluster
it does not skip file-downloading step
can we provide a jar path from hdfs ? while creating function
it never skip status of job-file-downloading because all my workers node are in the different machine
i have 4 machine my workers node at 2 and 4 mist at 1
if i create without deploy-mode cluster all workers node will be created at 1 which is same with mist so there is no problem with that case
Vadim Chelyshov
@dos65
Oh. Unfortunately, it impossible to skipfile-downloading step.
Hdsf paths support and other non-local path was removed in 1.0.0
Ali Emirhan Kurt
@AEKurt
actually skip is wrong word sorry for that i cannot pass the file-downloading step suddenly job is closed and there is no log about it
when i create function from local path, if i run job it distribute my jars all of other workers ?
Vadim Chelyshov
@dos65
Yes - worker node downloads jar from master node.
You need to check if mist master http port is accessible from worker nodes
Ali Emirhan Kurt
@AEKurt
what if i put my jar on every worker with the same path of master ?
can i speed up file-downloading process ?
Vadim Chelyshov
@dos65
It won't work out of the box without changing mist sources.
Ali Emirhan Kurt
@AEKurt
all of my workers reach master node
Vadim Chelyshov
@dos65
suddenly closed - does it complete with failure or success?
Ali Emirhan Kurt
@AEKurt
nothing just removed from ui
and also no log about it
spark-submit --deploy-mode cluster --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.mongodb.output.uri=mongodb://king1.local:27017,king2.esensi.local:27017/spark-solr?replicaSet=rs --conf spark.scheduler.mode=FAIR --conf spark.driver.memory=4g --conf spark.default.parallelism=8 --conf spark.driver.cores=8 --conf spark.mongodb.output.collection=queryResult --conf spark.sql.broadcastTimeout=36000 --conf spark.master=yarn --conf spark.executor.memory=8g --class io.hydrosphere.mist.worker.Worker /home/hdfs/mist/mist-1.1.2/mist-worker.jar --master 192.168.2.70:2551 --name Yarn2_b7540bb8-7084-4d30-9804-43dfb6fbfcc0-pool-2
this is what mist use while creating context
and that is my function
Language scala
Path /home/hdfs/mist/jobjars/bigjob.jar
Context Yarn2
Class com.king.scala.mist.MistFunction
i think worker nodes cannot download my bigjob.jar from master i dont know why
Vadim Chelyshov
@dos65
Could you check what happens in logs? It's strange that these jobs aren't presented on ui.
Ali Emirhan Kurt
@AEKurt
nothing in logs interesting.
artifact upload api has special feature ? i just only upload my jars while creating functions
Vadim Chelyshov
@dos65
@AEKurt There should be messages about job statuses: started, completed, failed ...
Even if job fails there should a message about it.
Ali Emirhan Kurt
@AEKurt
@dos65 after the status of queued its removed from the screen when i checked from yarn i can
see application_1558619048074_2698 hdfs io.hydrosphere.mist.worker.Worker SPARK default 0 Thu Jul 18 10:02:59 +0300 2019 N/A RUNNING UNDEFINED 3 3 6144 0 0 1.7 1.7
ApplicationMaster
if i kill from yarn i can show from ui and status is failed
Ali Emirhan Kurt
@AEKurt
do i need to make extra configuration for it ? runners etc
Mithun Raj Arackal
@mithunvillae
Is there any example available online for testing Mist functions in Scala? The links provided in the docs page is broken.
Ali Emirhan Kurt
@AEKurt
Mithun Raj Arackal
@mithunvillae
The link in the bottom of that page is not opening. But I think it should point to https://github.com/Hydrospheredata/mist/blob/master/examples/examples/src/test/scala/TestExampleSpec.scala
Ali Emirhan Kurt
@AEKurt
i can open the page that you sent
but when i click the link bottom of the page yes there is a problem
Vadim Chelyshov
@dos65
@mithunvillae you right - don't be afraid of sending a PR with fix)
Vadim Chelyshov
@dos65
Ok, links have been updated)
Ali Emirhan Kurt
@AEKurt
@dos65 ty
Vadim Chelyshov
@dos65
@AEKurt also, I made a new release - 1.1.3. It includes fixes for problem that you discovered
Ali Emirhan Kurt
@AEKurt
thank you if i discover anything else i will inform you
balauppalapati
@balauppalapati

Hi.. I have migrated my project to 2.12. I am using spark 2.4.0 with mist 1.1.3. I have created assembly jar and submitted to mist-cli. None of the jobs are being listed as functions in mist ui. Anamolies which i have noticed are:

  1. When I have submitted jar to mist-cli, only get url's are being listed. Earlier post url's along with corresponding inputs used to be listed.
  2. Found this type of error corresonding to each job in infoprovider.log,
    2019-07-30 20:37:17 INFO FunctionInfoProviderActor:107 - Responding with err on GetFunctionInfo(com.scienaptic.spark.jobs.PMMLEvaluatorJob$,/home/bala/Downloads/mist/mist-1.1.3/data/artifacts/iris_0.0.1.jar,pmml-evaluator,EnvInfo(PythonEntrySettings(python,python))): class java.lang.NoClassDefFoundError Could not initialize class com.scienaptic.spark.jobs.PMMLEvaluatorJob$
    where as the given job exists in the corresponding path.

Pretty much sure that conf is proper given similar config was working fine earlier. Is there anything which i am missing ?

Vadim Chelyshov
@dos65
Are you sure that you use mist and spark distros that were built for scala 2.12?
balauppalapati
@balauppalapati
To run mist locally, picked it from http://repo.hydrosphere.io/hydrosphere/static/mist-1.1.3.tar.gz
Similarly included mist 2.12 build in dependencies of project. I am skeptic about spark build. I tried with https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz.
Also tried spark build listed in mist release 1.1.0 - http://repo.hydrosphere.io/hydrosphere/static/spark/spark-2.4.0-bin-hadoop2.7-scala-2.12.tgz
Vadim Chelyshov
@dos65
balauppalapati
@balauppalapati
Ok. Will try with this build
Vadim Chelyshov
@dos65
And I'm not fully sure about spark build from repo.hydrosphere.io. I just built it from sources to check that mist works on 2.12. I can't remember if I tested python on it or not.
balauppalapati
@balauppalapati
Have tried it with above mist and spark build listed in release. It worked.
Thanks @dos65. I think it would be better to update docs for 2.12.
Vadim Chelyshov
@dos65
Great!
You right about docs, these things should be mentioned )