by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    J Khalaf
    @Ciwan1859
    Hi all
    I have a question and hope someone here can help.
    Say I have Kafka running on a machine. And the topic is just sat there idle with nothing happening. Will Kafka eventually kill that topic and discard whatever messages were sitting in the topic?
    SrinathJi
    @srinathji
    @Ciwan1859 It won't take any action until you delete the kafka topic
    ravi522
    @ravi522
    Hello sir,
    Faced some problem using Kafka in spring boot. When send a message to listener. listener listen message and done some validation on message and send back response like validation completed the message assign to new topic.But i want to listen the new topic. its taken some time to listen that topic. I hope you understand this question.
    Example:
    1) First create a Topic="user" send to Consumer. Consumer listen the topic using @Kafkalistener done some validation on user. Send back to new topic like topic="validation". This is the problem i want read this topic takes some time. like 100ms.
    dvirgiln
    @dvirgiln
    @dvirgiln
    Hi, i have a problem connecting with my producer to kafka, i receive this error:
    Failure(org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.)
    could it be related to the kafka listeners?
    I started kafka with the docker image: and this is the parameter i setup:
    KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
    Nick
    @GrigorievNick
    Hi
    Do this is good place to ask question about kafka-streams?
    Aditya Parikh
    @adityaparikh91087

    I had a question on Kafka Connect Single Message Transforms. I am using S3 as a sink. In most cases, I want to write the message as is. For certain messages with large payloads, the payload is omitted and I need to call a Rest API (Spring-Boot app) to get the payload and enrich the message being written to S3.
    Following along
    https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/ReplaceFieldTest.java

    I believe my logic should be here https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java#L142

    I kind of get what is going on here https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/ReplaceFieldTest.java#L43

    I couldn't find a simpler example to do so.

    Paul Snively
    @paul-snively
    Hi everyone. Has anyone else run into an issue where close on TopologyTestDriver in Kafka Streams doesn't unregister TestInputTopics? That is, I can do the new TopologyTestDriver(), createInputTopic("topic-1"), driver.close() dance once, but the next go-round, I get an exception saying topic "topic-1" was already registered by another source. My understanding is that close disposes of all state.
    Paul Snively
    @paul-snively
    Nevermind. This was a bug in my ScalaCheck Gen. :-(
    Paul Snively
    @paul-snively
    In Kafka Streams, does anyone know why:
    Topologies:
       Sub-topology: 0
        Source: KTABLE-SOURCE-0000000016 (topics: [KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000014-topic])
          --> KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000017
        Source: KSTREAM-SOURCE-0000000001 (topics: [input-topic-1])
          --> KTABLE-SOURCE-0000000002
        Processor: KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000017 (stores: [input-topic-1-STATE-STORE-0000000000])
          --> KTABLE-FK-JOIN-OUTPUT-0000000018
          <-- KTABLE-SOURCE-0000000016
        Processor: KTABLE-FK-JOIN-OUTPUT-0000000018 (stores: [])
          --> KTABLE-TOSTREAM-0000000020
          <-- KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-RESOLVER-PROCESSOR-0000000017
        Processor: KTABLE-SOURCE-0000000002 (stores: [input-topic-1-STATE-STORE-0000000000])
          --> KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000007
          <-- KSTREAM-SOURCE-0000000001
        Processor: KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000007 (stores: [])
          --> KTABLE-SINK-0000000008
          <-- KTABLE-SOURCE-0000000002
        Processor: KTABLE-TOSTREAM-0000000020 (stores: [])
          --> KSTREAM-SINK-0000000021
          <-- KTABLE-FK-JOIN-OUTPUT-0000000018
        Sink: KSTREAM-SINK-0000000021 (topic: output-topic)
          <-- KTABLE-TOSTREAM-0000000020
        Sink: KTABLE-SINK-0000000008 (topic: KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000006-topic)
          <-- KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000007
    
      Sub-topology: 1
        Source: KSTREAM-SOURCE-0000000004 (topics: [input-topic-2])
          --> KTABLE-SOURCE-0000000005
        Source: KTABLE-SOURCE-0000000009 (topics: [KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-0000000006-topic])
          --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000011
        Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000011 (stores: [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000010])
          --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000012
          <-- KTABLE-SOURCE-0000000009
        Processor: KTABLE-SOURCE-0000000005 (stores: [input-topic-2-STATE-STORE-0000000003])
          --> KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000013
          <-- KSTREAM-SOURCE-0000000004
        Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000012 (stores: [input-topic-2-STATE-STORE-0000000003])
          --> KTABLE-SINK-0000000015
          <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000011
        Processor: KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000013 (stores: [KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000010])
          --> KTABLE-SINK-0000000015
          <-- KTABLE-SOURCE-0000000005
        Sink: KTABLE-SINK-0000000015 (topic: KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-0000000014-topic)
          <-- KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000012, KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000013
    would be invalid? Visualizing it at https://zz85.github.io/kafka-streams-viz/, it seems OK to me. But in a test, I get no output on output-topic.