by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Maxime Hurtrel
    @crazyman_twitter
    Welcome everyone :) You can find more information and the first documentation for IOStream at the bottom of this page : https://labs.ovh.com/iostream
    Collignon-Ducret Rémi
    @miton18
    Hi !
    Steven Le Roux
    @StevenLeRoux
    Thanks @crazyman_twitter !
    R
    @rveyrin_twitter
    thanks maxime. What could be nice (general to all the labs feature) is to have a theoritical roadmap (alpha/beta/production, with a confidence score) and a theoritical target price
    Steven Le Roux
    @StevenLeRoux
    @rveyrin_twitter Thanks for your feedback. I agree with you and since it's rather clear for us we should make it public very soon. For the target price, we will tease it here soon, but for now, let's say it will be very well positionned regarding the competition.
    R
    @rveyrin_twitter
    thanks Steven to have take it in consideration. The scale/probability of production and price range could help your customers to go a bit further than an hello world test
    Micaël Pais Novo
    @micael-pn
    Hello
    We are interrested in this product. Here is our use case : we have mysql databases hosted on public cloud instances. We would like to monitor databases changes with some tool (CDC Debezium Connector seems to do what we want : https://pulsar.apache.org/docs/en/io-cdc-debezium/). Is it possible to achieve what we need with ioStream plateform ? Moreover, is ioStream compatible with OVH vrack ? It would be great to keep producers and consumers only on a private network.
    Thanks !
    Steven Le Roux
    @StevenLeRoux
    Hi @micael-pn , It is definitely possible to achieve it. For the moment, Pulsar IO is not included in the managed service, but it will be. For now, you could plug a CDC agent on your database and push events to your topic. We're actively working on the Kafka API support, so you would end up with capabilities like Kafka Connect and particularly Debezium. For example : https://debezium.io/documentation/reference/0.10/architecture.html
    Steven Le Roux
    @StevenLeRoux
    Also while Kafka is not publicly opened, it seems you could push directly from Debezium to Pulsar: https://github.com/debezium/debezium-examples/tree/master/apache-pulsar
    We also have a CDC agent that support most databases and can push to Pulsar natively. Maybe @Pirionfr could tell more on it? :)
    Micaël Pais Novo
    @micael-pn
    Thanks for your answers
    I see that @rveyrin_twitter asked for thorical roadmap and pricing. we woul also be interested in this if you have any news (for Pulsar IO Support, vrack, ...)
    If you have any details about your CDC agent and how we could use it for our use case, I would apreciate @Pirionfr :)
    Steven Le Roux
    @StevenLeRoux
    @micael-pn yes we're looking for a tool to communicate the roadmap :) The Beta is also there so that you can feed us with all your expectations regarding the product (Pulsar IO and vRack support are good examples).
    erapx
    @erapx
    Hello, is there currently a limit on the number of consumers that can attach to a shared subscription? I see some HTTP 429 connection errors
    Florentin DUBOIS
    @FlorentinDUBOIS
    Hi @erapx, could you send me your service name in private message? I will check what happened
    erapx
    @erapx
    @FlorentinDUBOIS sure, doing that now
    info2000
    @info2000
    there's some documentation about iostream? what is the in-memory capacity?
    Pierre Zemb
    @PierreZ
    Hi @info2000! You can find more information about ioStream here (https://labs.ovh.com/iostream), and the documentation is over here (https://labs.ovh.com/iostream/documentation). For the in-memory part, as we are relying on apache pulsar underneath ioStream, you can create non-persistent topics, i.e Non-persistent topics are Pulsar topics in which message data is never persistently stored and kept only in memory.
    info2000
    @info2000
    thanks @PierreZ on my dashboard don't show the retention period and the memory size, in other words: can I produce/send messages up to 1 terabyte without consum them?
    Pierre Zemb
    @PierreZ
    For persistent topic, you can configure the backlog retention (retention for unasked messages) and the replay retention (retention for acked messages) through the manager. For non-persistent topic, there is no retention: if you have no application reading the data you are pushing, the messages are discarded. In order to keep messages without consuming them, you need to set the backlog retention for a persistent topic. We are not exposing a retention based on volume at the current state of the beta, rather a time-based retention (X hours).
    Micaël Pais Novo
    @micael-pn
    Hello
    I have been playing around with ioStream and Apache Pulsar CDC for mysql (documentation : https://pulsar.apache.org/docs/en/io-cdc-debezium/). I tried to configure debezium to monitor mysql changes to push them to pulsar. I get the following error : Failed to get partitioned topic metadata: org.apache.pulsar.client.api.PulsarClientException$LookupException: java.lang.IllegalAccessException: Don't have permission to administrate resources on this tenant
    Has anyone achieved to setup something similar with ioStream ? I would really appreciate some feedback and help. Obviously, I can provide more info if needed.
    Thanks !
    Steven Le Roux
    @StevenLeRoux
    Hello @micael-pn ! welcome on ioStream :)
    did you create the topic before?
    Maybe debezium is trying to create it but we don’t expose the admin API yet
    you need to use the ovh api or UI manager
    Pierre Zemb
    @PierreZ
    Hell @micael-pn ! I think you are trying to deploy a Pulsar IO connectors. For the moment, Pulsar IO is not included in the managed service, but it will be. For now, you could plug a CDC agent on your database and push events to your topic. We're actively working on the Kafka API support, so you would end up with capabilities like Kafka Connect and particularly Debezium.
    Micaël Pais Novo
    @micael-pn
    @StevenLeRoux Can you specify how can I create a new topic (https://pulsar.apache.org/docs/en/concepts-messaging/#topics). Is it a "Stream" in the UI manager ? If so, yes I created a topic before.
    @PierreZ Well I tried to deploy Pulsar IO connector (configuring CDC agent to push events to the ioStream instead of pushing locally) and Debezium from the related documentation (https://debezium.io/documentation/reference/0.10/operations/embedded.html). No luck for the moment. But I assume that you are saying that there is no way to stream Mysql changes to ioStream for the moment until ioStream supports Kafka API ?
    Pierre Zemb
    @PierreZ
    Yes, a stream is equivalent to a topic. Pulsar IO connectors are running inside a Pulsar cluster, and it is disabled on ioStream for now. We will work on Pulsar IO support after the Kafka support
    Micaël Pais Novo
    @micael-pn
    Hello
    Any update or information regarding the ioStream roadmap ?
    • What will be the pricing ?
    • When will it be production ready ?
    • When Kafka endpoint will be available ?
    • What about Pulsar connectors ?
      Thank you :)
    Steven Le Roux
    @StevenLeRoux
    Hi @micael-pn ! Thanks for your interest !
    The pricing is being finalising, but we're thinking about something like : for a monthly plan : Bandwidth : 0.05€ / GB
    Storage : 0.20€ / GB
    Steven Le Roux
    @StevenLeRoux
    Kafka protocol has been merged on the broker side, but as product, we use a proxy in front, but there is still a bit of work to get it ready for prime time. We will need to battle test it so that the customer experience will be awesome, so I guess not before a couple of month. It's not very precise, because there is still some unclear area on the implementation strategy, but we will get there :)
    Steven Le Roux
    @StevenLeRoux
    We also want Kafka as a protocol to be part of the main feature to release the product as GA. I think we will reach GA at the end of FY20. Still we plan to add the connector features before, around summer.
    What are you interested in? what do you expect as basic feature for a topic as a service product?
    Micaël Pais Novo
    @micael-pn
    Hi
    Thank you for your answer !
    As mentioned earlier in this thread, our main use case would be to stream changes from our main Mysql database to ioStream so that other internal services connected in the same vRack would consume those events to trigger some automated actions. That's why we are interested in both Kafka endpoint (to connect debezium CDC for example) or embedded Pulsar connectors.
    If someone has any idea how we can achieve our use case with ioStream, I would be happy to discuss about that !
    Pierre Zemb
    @PierreZ
    Hi Micael! I think you can do it by hand by deploying debezium and using kafka's endpoint when we will release it. As Steven said, we want to provide managed connectors, so the overall experience will be better after we added some managed connectors in our product :)
    Micaël Pais Novo
    @micael-pn
    So we must wait this summer to use managed connector planned this summer or wait the GA release planned before the end of the year to use ioStream for our use case ?
    Vincent Dieulesaint
    @vdieulesaint
    Hello everybody, I have just enrolled on ioStream. The kafka's endpoint is not available. Could you let me know when it will available ? Thks. Vincent
    Pierre Zemb
    @PierreZ

    Hi Vincent!

    as ioStream is a beta for now, we are stabilizing both Pulsar’s infrastructure and Kafka’s support for now. kafka is definitely an important milestone in our roadmap but we dont have a date for the release yet.
    Pierre

    Vincent Dieulesaint
    @vdieulesaint
    ok. let me know when you have an ETA (even in beta). We, at eco counter, are definitely interested by what you are doing right now.
    Pierre Zemb
    @PierreZ
    For sure I will let you know :)
    Philippe Sobczak
    @philso
    Hi all ! Very interested to use ioStream to build a live scores feed : Single Producer in France / Multiple consumers worldwide. We need to achieve the least possible latency. Do you have any roundtrip latency benchmarks available ? Thanks !!!
    Maxime Hurtrel
    @crazyman_twitter

    Hi dear IOstream LAB testers.

    This experience has been a great opportunity to express our interest in Apache Pulsar. However, the limited success of the beta service — and other strategic focuses — have resulted in us taking the very difficult decision to close it. An offcial email is currently being sent to all testers.

    The “IOstream” section of the OVHcloud Control Panel will be deactivated in early September. All the data and API endpoints associated to the service will be deleted, in accordance with OVHcloud beta terms, on the 14th September.

    We remain committed to open source, and were happy to contribute to the Apache Pulsar ecosystem through our contributions on the Kafka-on-Pulsar project https://github.com/streamnative/kop

    If you are interested in deploying your own Apache Pulsar on OVHcloud, please follow this official documentation: https://github.com/apache/pulsar/blob/master/site2/docs/deploy-kubernetes.md, that can help you deploy Pulsar on our managed Kubernetes service: https://www.ovhcloud.com/fr/public-cloud/kubernetes/.

    We are also exploring new approaches to offer alternative managed event services on OVHcloud in the mid-term future.

    Thanks again for your feedback in the name of all the team, and especially @PierreZ and @StevenLeRoux leads on the program