Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 01 2020 10:26
    raushni5 edited #560
  • Dec 01 2020 06:45
    raushni5 edited #560
  • Dec 01 2020 06:44
    raushni5 opened #560
  • May 14 2020 08:21
    Avik1993 commented #523
  • May 09 2020 18:45
    Avik1993 commented #523
  • May 09 2020 18:44
    blvp commented #523
  • May 09 2020 18:43
    blvp commented #523
  • May 09 2020 17:33
    Avik1993 commented #523
  • May 09 2020 17:20
    blvp commented #523
  • May 09 2020 12:14
    Avik1993 commented #523
  • Apr 07 2020 15:20
    blvp commented #559
  • Apr 07 2020 11:35
    amorskoy closed #559
  • Apr 07 2020 11:35
    amorskoy commented #559
  • Apr 07 2020 10:21
    blvp commented #559
  • Apr 07 2020 10:08
    amorskoy commented #559
  • Apr 07 2020 10:08
    amorskoy commented #559
  • Apr 06 2020 10:42
    amorskoy opened #559
  • Mar 26 2020 12:49
    blvp commented #558
  • Mar 26 2020 12:42
    kangtiann opened #558
  • Mar 26 2020 10:43
    pmiyandada commented #557
Leonid
@leonid133
Hi @Braffolk, I'll try to build
Braffolk
@Braffolk
thatd be really appreciated
Leonid
@leonid133
Hi @Braffolk
I reproduced your error,
So, for fast solution you can add
"org.glassfish.jersey.inject" % "jersey-hk2" % “2.30"
in mist.sbt (libraryDependencies )
So looks like we should prepare new version with maintain Spark 3.0 :) I’ll push ‘draft' branch for start. (draft/spark-3.0.1 )
Braffolk
@Braffolk
will try it out, thanks
image.png
now getting this. testing on aws
Braffolk
@Braffolk
could be an issue with the old docker-java version
Leonid
@leonid133
oh, yep. I'll take a look. It works for workers.runner = "local", you can use
sbt -DsparkVersion=3.0.1 -DscalaVersion=2.12.7 mistRun
or change to workers.runner = "local" in configs/docker.conf
Braffolk
@Braffolk
ahh, were using local[*] right now with 5 workers
Leonid
@leonid133
did you mean mist.context-defaults.spark-conf.spark.master=local[*] ?
There are also several ways how mist spawn workers
mist.workers.runner
local - just local submit-submit on the same node
docker - Spawn worker in separated docker container
manual - custom script for worker spawning (https://github.com/Hydrospheredata/mist/blob/master/examples/misc/manual_worker_start.py)
Braffolk
@Braffolk
yes, thats what i meant. And that runs as seprate docker containers
Braffolk
@Braffolk
wish i could fix it myself, but im not familiar with scala D: mainly write kotlin and python.
Braffolk
@Braffolk
any updates?
Leonid
@leonid133
Hi @Braffolk, I reproduced your error, looks like you do not attach volume with docker socket (this should help -v /var/run/docker.sock:/var/run/docker.sock)
docker run -p 2004:2004 -v /var/run/docker.sock:/var/run/docker.sock hydrosphere/mist:1.1.3-3.0.1-scala-2.12
Braffolk
@Braffolk
huge thanks!
i finally got everything working
our data pipeline seems to work now using spark 3.0.1, had to make changes to max artifact size locally though
since our artifact is currently larger than 250mb
Braffolk
@Braffolk
will try to get everything up and running in AWS now. I can share the updated cloudformation Spark 3.0.1 install script if anyone needs it
Braffolk
@Braffolk
i have working docker image on dockerhub at braffolk/mist:1.1.3-3.0.1-scala-2.12
Braffolk
@Braffolk
odd. Whats FunctionInfoProvider for?
2020-11-02 15:36:06 INFO DbMigrate:44 - Successfully applied 2 migrations to schema "PUBLIC" (execution time 00:00.083s). 2020-11-02 15:36:06 INFO MasterServer$:270 - DB started 2020-11-02 15:36:06 INFO MasterServer$:270 - LogsSystem started 2020-11-02 15:36:27 ERROR RestartSupervisor:159 - Starting child for FunctionInfoProvider failed java.lang.IllegalStateException: Initialization of FunctionInfoProvider failed of timeout at io.hydrosphere.mist.master.jobs.ActorRefWaiter$IdentityActor$$anonfun$receive$1.applyOrElse(FunctionInfoProviderRunner.scala:60) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at io.hydrosphere.mist.master.jobs.ActorRefWaiter$IdentityActor.aroundReceive(FunctionInfoProviderRunner.scala:48) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) at akka.actor.ActorCell.invoke(ActorCell.scala:496)
on AWS its unable to find FunctionInfoProvider
Braffolk
@Braffolk
oh, looked into the infoprovider log. For the local one it warns about not being able to load native hadoop library, then starts slf4jlogger and remote, however on the aws instance it spams unable to load native library for 10 times and thats the end of output, hmm.
Braffolk
@Braffolk
could this be related? "By some reasons, there is some problems with running Mist right after intance was started. InfoProvider can't connect to master"
Braffolk
@Braffolk
solved. used the mist 1.1.3 version without scala 2.12.
Braffolk
@Braffolk
also created an image for spark with hadoop 3.2
braffolk/mist:1.1.3-3.0.1-scala-2.12-hadoop3.2
hadoop 2.7 is broken when reading from AWS S3 buckets using s3a, so its good to have this too
Raushni Khan
@raushni5
Hey..
I am trying to cancel a long running spark job using mist..however mist continues to display this job in cancelling state which eventually throws a something went wrong error..can somebody please help out on this?
1 reply
Braffolk
@Braffolk
@raushni5 we're also having this issue. From time to time our Mist instance seems to go into an invalid state where all cancelling jobs get stuck. So far we've been postponing finding out the reason as we're working on other things, but once ill get to the issue, ill defo post on possible solutions here
Unless you solve it first. Would be great to hear the fix then
Braffolk
@Braffolk

Anyone seen an error like this? This happens using local[*] mode and happens during worker start and the start fails with this error:

Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of com.github.dockerjava.api.command.InspectContainerResponse out of START_ARRAY token

this happens in our ECS AWS cluster inside a docker instance that starts docker instances as workers, if that matters
worked fine when we just started a cloudformation instancce with a .sh install script
Braffolk
@Braffolk
oddly enough, its attempting to inspect a docker container with the id "" that doesnt actually exist
4 replies
Braffolk
@Braffolk
If anyone needs newer hadoop versions or spark 3.0, we have a working fork over here until the main repo gets updated: https://github.com/kaitumisuuringute-keskus/mist
Braffolk
@Braffolk
hello, are there any examples of using mist with Kafka or MQTT?
Pavel Borobov
@blvp
Hi, here is a link for reference https://hydrosphere.io/mist-docs/reactive_api.html
You can also find example in source code with python mqtt example, not sure how old is it, so be careful :
Braffolk
@Braffolk

Hi! Have there been any plans to make the reactive api support streaming individual dataset rows? Or also support spark streaming? In some larger (timeline for example) calculations some of the data starts finishing quite fast and hence could be sent down to be processed. Right now the reactive API does support lifecycle events, but not this.

I'm solving this myself by creating another RSocket based server running on the mist master instance where these rows are written to and to which other microservices can subscribe to. This isn't very optimal as the original reactive api could support this much more seamlessly.

sinbadonline
@sinbadonline
@Braffolk Hi, do you know the reason for the issue "ERROR t.utils.akka.RestartSupervisor:159 Starting child for FunctionInfoProvider failed"? Saw you met this issue months ago
Braffolk
@Braffolk
@sinbadonline hmm, late reply, but ill try to remember
i think that was an issue with the scala version
this works for me docker run -p 2004:2004 -v /var/run/docker.sock:/var/run/docker.sock kaitumisuuringutekeskus/mist:1.1.3-3.0.1-scala-2.12-hadoop3.2
Braffolk
@Braffolk
image.png
Anyone seen something like this? It inconsistently started happening on some jobs and im not sure what could be causing it
i guess i could just increase the stack size
but maybe there could be an underlying reason