Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jean-Baptiste Note
    @jbnote
    Hi guys, is there a design document that I could contribute to for Koya ?
    I'm poised to deploy kafka in production with slider and I had to solve a number of problems and would like to share/compare with the design decisions you took.
    Thomas Weise
    @tweise
    Jean, there are the project README and the presentation from the Hadoop Summit in Brussels: http://www.slideshare.net/Hadoop_Summit/kafka-on-yarn-koya-an-open-source-initiative-to-integrate-kafka-yarn
    Do you have something in .md format that you would like to contribute to the project? Pull requests are welcome :smile: Would be happy to learn from your setup and how we can potentially enhance our approach.
    Jean-Baptiste Note
    @jbnote
    Not in md format yet, but will do. I do have a pending MR for configuration flexibility though :)
    Thomas Weise
    @tweise
    Yes, we are looking at the PR. Thinking of combining both approaches - your XML file template and option to specify keys in appConfig.json. We are planning to support the Confluent package also, which has components other than broker.
    Jean-Baptiste Note
    @jbnote
    Well the whole point is precisely to be able to override from the appConfig. You can override any key in the XML file with the appConfig, or add any specific key in the appConfig, they'll be taken into account in the generated file.
    Siyuan Hua
    @siyuanh
    I totally got your point, my point is the XML template seems useless.
    Jean-Baptiste Note
    @jbnote
    The XML template is not useless because it allows you to bounce some defaults to some parameters, ${@//site.xxxx} which can only dynamically be resolved (ie at component instance time rather than app time). And i'd rather do this in an XML file than in the python script; it's much more clear to do it there, because it's quite clear that you're having data there. Have a look at other packaged apps (hbase, accumulo, and al) it seems to me that this is how it was designed to be used.
    Thomas Weise
    @tweise
    Support for reading the component configuration was committed in Slider: https://issues.apache.org/jira/browse/SLIDER-812
    We should also add support for it.
    Jean-Baptiste Note
    @jbnote
    @tweise: I certainly can try this, but can you specify what we'd need to pass through this mechanism ?
    lowchinwei
    @lowchinwei
    Hi,
    I am testing with koya and notice that the kafka log dir is assigned to ${AGENT_WORK_ROOT}/app/run. So, if I stop koya and start using slider, the previous log is gone.
    So, what is the proper configuration for this?
    alancnet
    @alancnet
    Is koya still alive and kicking? Would it be suitable as a streaming data store for production systems? I haven't seen anything since 2015
    Siyuan Hua
    @siyuanh
    @alancnet We are not actively working on koya right now. Hopefully the project can be merged into slider (https://slider.incubator.apache.org/index.html) If you are interested you can take a look and contribute to this
    Thomas Weise
    @tweise
    @alancnet KOYA has been merged into Slider and the new locations is https://github.com/apache/incubator-slider/tree/develop/app-packages/kafka