Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 01 2020 10:26
    raushni5 edited #560
  • Dec 01 2020 06:45
    raushni5 edited #560
  • Dec 01 2020 06:44
    raushni5 opened #560
  • May 14 2020 08:21
    Avik1993 commented #523
  • May 09 2020 18:45
    Avik1993 commented #523
  • May 09 2020 18:44
    blvp commented #523
  • May 09 2020 18:43
    blvp commented #523
  • May 09 2020 17:33
    Avik1993 commented #523
  • May 09 2020 17:20
    blvp commented #523
  • May 09 2020 12:14
    Avik1993 commented #523
  • Apr 07 2020 15:20
    blvp commented #559
  • Apr 07 2020 11:35
    amorskoy closed #559
  • Apr 07 2020 11:35
    amorskoy commented #559
  • Apr 07 2020 10:21
    blvp commented #559
  • Apr 07 2020 10:08
    amorskoy commented #559
  • Apr 07 2020 10:08
    amorskoy commented #559
  • Apr 06 2020 10:42
    amorskoy opened #559
  • Mar 26 2020 12:49
    blvp commented #558
  • Mar 26 2020 12:42
    kangtiann opened #558
  • Mar 26 2020 10:43
    pmiyandada commented #557
Pavel Borobov
@blvp
@kusumakarb hi there, sorry for late response. I think we're open for any kind of contribution and I can kindly guide you through this process if needed. One thing to pay attention is backward compatibility for this feature. We need to have previous behaviour if nothing configured and we can configure topic separation as you requested. Please feel free to reach out to me for any questions.
kusumakar
@kusumakarb
@blvp No problem. Thanks for the response here. I've sent you a DM on gitter for the same.
Braffolk
@Braffolk
hey, how does Mist deal with higher loads? I have a one machine setup on AWS using their new r6g instances. Launching a hundred or two hundred jobs at the same time is fine, but at around 400 it simply crashes and gives error 500 from both the websocket and the UI stops loading. Ive seen it recover once from the error 500, but usually its dead after that
Pavel Borobov
@blvp
Hello there! We didn't run such workloads for this simple service, usually jobs are long running and queuing mechanism helps to run through them, but sure you need to pay attention to memory limits set for Java process. Could you please share the mist master and context configuration? As well as the run info of java process for master instance? Running workers in local node can also cause this outage.
Braffolk
@Braffolk
anyone had any luck getting mist to work with spark 3? I managed to build it and now im trying to debug the image. doesnt seem to be running even though the console says it is running
nvm! its up and running on localhost
time to see if it actually works
Braffolk
@Braffolk
hm, getting "java.lang.IllegalStateException: InjectionManagerFactory not found." when its attempting to start a worker
Braffolk
@Braffolk
seems tobe an issue with the DockerStarter and jersey. Not sure how to fix it
Leonid
@leonid133
Hi @Braffolk, I'll try to build
Braffolk
@Braffolk
thatd be really appreciated
Leonid
@leonid133
Hi @Braffolk
I reproduced your error,
So, for fast solution you can add
"org.glassfish.jersey.inject" % "jersey-hk2" % “2.30"
in mist.sbt (libraryDependencies )
So looks like we should prepare new version with maintain Spark 3.0 :) I’ll push ‘draft' branch for start. (draft/spark-3.0.1 )
Braffolk
@Braffolk
will try it out, thanks
image.png
now getting this. testing on aws
Braffolk
@Braffolk
could be an issue with the old docker-java version
Leonid
@leonid133
oh, yep. I'll take a look. It works for workers.runner = "local", you can use
sbt -DsparkVersion=3.0.1 -DscalaVersion=2.12.7 mistRun
or change to workers.runner = "local" in configs/docker.conf
Braffolk
@Braffolk
ahh, were using local[*] right now with 5 workers
Leonid
@leonid133
did you mean mist.context-defaults.spark-conf.spark.master=local[*] ?
There are also several ways how mist spawn workers
mist.workers.runner
local - just local submit-submit on the same node
docker - Spawn worker in separated docker container
manual - custom script for worker spawning (https://github.com/Hydrospheredata/mist/blob/master/examples/misc/manual_worker_start.py)
Braffolk
@Braffolk
yes, thats what i meant. And that runs as seprate docker containers
Braffolk
@Braffolk
wish i could fix it myself, but im not familiar with scala D: mainly write kotlin and python.
Braffolk
@Braffolk
any updates?
Leonid
@leonid133
Hi @Braffolk, I reproduced your error, looks like you do not attach volume with docker socket (this should help -v /var/run/docker.sock:/var/run/docker.sock)
docker run -p 2004:2004 -v /var/run/docker.sock:/var/run/docker.sock hydrosphere/mist:1.1.3-3.0.1-scala-2.12
Braffolk
@Braffolk
huge thanks!
i finally got everything working
our data pipeline seems to work now using spark 3.0.1, had to make changes to max artifact size locally though
since our artifact is currently larger than 250mb
Braffolk
@Braffolk
will try to get everything up and running in AWS now. I can share the updated cloudformation Spark 3.0.1 install script if anyone needs it
Braffolk
@Braffolk
i have working docker image on dockerhub at braffolk/mist:1.1.3-3.0.1-scala-2.12
Braffolk
@Braffolk
odd. Whats FunctionInfoProvider for?
2020-11-02 15:36:06 INFO DbMigrate:44 - Successfully applied 2 migrations to schema "PUBLIC" (execution time 00:00.083s). 2020-11-02 15:36:06 INFO MasterServer$:270 - DB started 2020-11-02 15:36:06 INFO MasterServer$:270 - LogsSystem started 2020-11-02 15:36:27 ERROR RestartSupervisor:159 - Starting child for FunctionInfoProvider failed java.lang.IllegalStateException: Initialization of FunctionInfoProvider failed of timeout at io.hydrosphere.mist.master.jobs.ActorRefWaiter$IdentityActor$$anonfun$receive$1.applyOrElse(FunctionInfoProviderRunner.scala:60) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at io.hydrosphere.mist.master.jobs.ActorRefWaiter$IdentityActor.aroundReceive(FunctionInfoProviderRunner.scala:48) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) at akka.actor.ActorCell.invoke(ActorCell.scala:496)
on AWS its unable to find FunctionInfoProvider
Braffolk
@Braffolk
oh, looked into the infoprovider log. For the local one it warns about not being able to load native hadoop library, then starts slf4jlogger and remote, however on the aws instance it spams unable to load native library for 10 times and thats the end of output, hmm.
Braffolk
@Braffolk
could this be related? "By some reasons, there is some problems with running Mist right after intance was started. InfoProvider can't connect to master"
Braffolk
@Braffolk
solved. used the mist 1.1.3 version without scala 2.12.
Braffolk
@Braffolk
also created an image for spark with hadoop 3.2
braffolk/mist:1.1.3-3.0.1-scala-2.12-hadoop3.2
hadoop 2.7 is broken when reading from AWS S3 buckets using s3a, so its good to have this too
Raushni Khan
@raushni5
Hey..
I am trying to cancel a long running spark job using mist..however mist continues to display this job in cancelling state which eventually throws a something went wrong error..can somebody please help out on this?
1 reply
Braffolk
@Braffolk
@raushni5 we're also having this issue. From time to time our Mist instance seems to go into an invalid state where all cancelling jobs get stuck. So far we've been postponing finding out the reason as we're working on other things, but once ill get to the issue, ill defo post on possible solutions here
Unless you solve it first. Would be great to hear the fix then
Braffolk
@Braffolk

Anyone seen an error like this? This happens using local[*] mode and happens during worker start and the start fails with this error:

Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of com.github.dockerjava.api.command.InspectContainerResponse out of START_ARRAY token

this happens in our ECS AWS cluster inside a docker instance that starts docker instances as workers, if that matters
worked fine when we just started a cloudformation instancce with a .sh install script
Braffolk
@Braffolk
oddly enough, its attempting to inspect a docker container with the id "" that doesnt actually exist
4 replies
Braffolk
@Braffolk
If anyone needs newer hadoop versions or spark 3.0, we have a working fork over here until the main repo gets updated: https://github.com/kaitumisuuringute-keskus/mist
Braffolk
@Braffolk
hello, are there any examples of using mist with Kafka or MQTT?
Pavel Borobov
@blvp
Hi, here is a link for reference https://hydrosphere.io/mist-docs/reactive_api.html
You can also find example in source code with python mqtt example, not sure how old is it, so be careful :