These are chat archives for spring-cloud/spring-cloud

15th
Sep 2016
bitsofinfo
@bitsofinfo
Sep 15 2016 04:19

I have one kafka topic "my-event-topic" which has events of type "MyEvent" published to it.

I have one particular spring-cloud appX (3 instances) which needs to consume from this topic and update a database.

All "appX" instances have this property defined for their consumer group: spring.cloud.stream.bindings.my-event-topic.group: appX-group1 so that only one live instance will process the event and update the shared DB. Each "appX" instance is using @StreamListenerbound to this channel/topic name.

What I would also like to do is also have each "appX" instance have a separate @StreamListener that listens to the same topic "my-event-topic", however all 3 "appX" instances would receive it (not just one as w/ the consumer group). Is this possible? Or do I have to have a completely separate topic (i.e. "my-event-topic-copy1") to achieve this?

Jimmy K. Oak
@jimmyoak
Sep 15 2016 07:05
@bitsofinfo Have you tried to define another SubscribableChannel in your interface with a different name, but in config put the same destination topic and another group name? This way another consumer group should be created, consuming separately all events.
Something like:
public interface SampleChannelInput {
  String SAMPLE = "sample-channel-input";
  String ANOTHER_SAMPLE = "another-sample-channel-input";

  @Input(SAMPLE)
  SubscribableChannel input();

  @Input(ANOTHER_SAMPLE)
  SubscribableChannel anotherInput();
}
spring:
  cloud:
    stream:
      bindings:
        sample-channel-input:
          group: sample_group
          destination: "sample.channel"
          content-type: text/plain
        another-sample-channel-input:
          group: another_sample_group
          destination: "sample.channel"
          content-type: text/plain
Or if you want that all instances process the same message don’t put any group in another-sample-channel-input
Jimmy K. Oak
@jimmyoak
Sep 15 2016 07:40
Is someone using spring cloud stream kafka with autoscaling groups in AWS or similar over here? I cannot figure out how many instances I will have so I cannot put instanceCount and instanceIndex in config. Without it, multiple consumers consume the same message. Is there another way to achieve non-duplicates?
Ivan
@ismorodin
Sep 15 2016 08:06
How to exclude the services from ribbon balancer by tag name in consul ?
Marcin Grzejszczak
@marcingrzejszczak
Sep 15 2016 10:30
I'm no expert in comparison to Spencer but I guess you'd have to create a customIRule that would check the tag
espcially Customizing the Ribbon Client ... sections @ismorodin
Jimmy K. Oak
@jimmyoak
Sep 15 2016 10:41
Is there any way to have multiple instances consuming from kafka without setting instanceCount and instanceIndex? Without them I only get duplicated messages
Dave Syer
@dsyer
Sep 15 2016 11:09
That is correct
Or rather it is if you have a consumer group
Might be best to wait till @mbogoevici wakes up and ask him though
Jimmy K. Oak
@jimmyoak
Sep 15 2016 11:23
Yep. I set a group. But I’ve seen that spring-cloud-stream-kafka uses a SimpleConsumer and tries to manage the consumer group and partitions on its way, instead of using a Consumer from kafka-clients lib which leaves Kafka deal with that.
In a SimpleConsumer from kafka-clients lib you cannot put a group id
I’ll wait for @mbogoevici :)
Odín del Río
@odin-delrio
Sep 15 2016 11:47
I'm with @jimmyoak looking for a solution here, we saw the "instanceCount" and "instanceIndex" options to allow multiple consumers to read from the same topic, but that solution will make our deploys harder... and since kafka can deal with consumer groups, we are pretty sure that it has to be a way to use spring-cloud-stream without set that config properties... are we right?
Marius Bogoevici
@mbogoevici
Sep 15 2016 13:27
@odin-delrio That was true for version 1.0 with Kafka 0.8 (since we didn't use the high-level consumer) Since the Brooklyn release train we support Kafka 0.9 and the new consumer - so you can do away with those properties. You need to set a consumer group though
If you were to use Spring Cloud Data Flow though, those properties would be set automatically.
@jimmyoak ^^
Jimmy K. Oak
@jimmyoak
Sep 15 2016 13:30
We don’t use SpringCloudDataFlow :( And we can’t change all deploy flow
(I tried to write a sad emoji, not a worried one)
We will try with the RC2 if it is publicly available
I’ve been looking on github repository of the kafka binder and I found the auto-rebalance-enabled property.
Marius Bogoevici
@mbogoevici
Sep 15 2016 13:33
yeah
I would recommend you use the Brookyn.RC1 BOM for version management
Dave Syer
@dsyer
Sep 15 2016 13:34
Or Camden.RC1 if you are using other spring cloud stuff
Marius Bogoevici
@mbogoevici
Sep 15 2016 13:34
as @dsyer said :)
Marius Bogoevici
@mbogoevici
Sep 15 2016 13:47
auto-rebalance-enabled is useful mostly for partitioned consumers
bitsofinfo
@bitsofinfo
Sep 15 2016 13:58
@jimmyoak ok, I'll try that. When I read the docs I was under the assumption that the names "sample-channel-input" and "another-sample-channel-input" were the names of topics in kafka, implying that you needed yet another topic to acheive this, or am I wrong?
I guess that is the name of the logical spring-cloud-stream "channel" and that "destination" is the literal physical one. Must be convention to use the channel name as the topic/dest name, if not explicity defined, correct?
Jimmy K. Oak
@jimmyoak
Sep 15 2016 14:25
@mbogoevici Is the Camden version compatible with Kafka 0.8 in some way?
Marius Bogoevici
@mbogoevici
Sep 15 2016 14:25
@jimmyoak no
The 0.9 client itself is not compatible and we decided that 0.8 was old enough not to warrant the effort to retrofit everything using the SimpleConsumer
Marius Bogoevici
@mbogoevici
Sep 15 2016 14:37
@bitsofinfo "I guess that is the name of the logical spring-cloud-stream "channel" and that "destination" is the literal physical one. Must be convention to use the channel name as the topic/dest name, if not explicity defined, correct?” -> yes
bitsofinfo
@bitsofinfo
Sep 15 2016 14:37
cool, thanks
Amin Abu-Taleb
@aabutaleb
Sep 15 2016 14:41
Hi everyone, dunno if this is the best place to drop this but I have an issue with Turbine that's driving me nuts, posted it in SO, any help will be very much appreciated: http://stackoverflow.com/questions/39511244/turbine-instanceurlsuffix-not-working
Spencer Gibb
@spencergibb
Sep 15 2016 14:42
it’s in my browser to look at
the latest release is Brixton.SR5 btw
Amin Abu-Taleb
@aabutaleb
Sep 15 2016 14:43
many thanks, let me know if you have any questions
newer that RELEASE?
Spencer Gibb
@spencergibb
Sep 15 2016 14:43
yes we’ve had 5 releases since
Amin Abu-Taleb
@aabutaleb
Sep 15 2016 14:43
jeez, you guys work fast xD
might that solve my issue?
Spencer Gibb
@spencergibb
Sep 15 2016 14:44
possibly, if not it’s helpful to debug on the latest code
Amin Abu-Taleb
@aabutaleb
Sep 15 2016 14:46
is there any open bug on Turbine related to my post that you might know of? I couldn't find anything
Spencer Gibb
@spencergibb
Sep 15 2016 14:54
not that I know of
Amin Abu-Taleb
@aabutaleb
Sep 15 2016 14:55
I updated turbine service to latest version, still failing. Trying to update zuul and eureka as well, and see if that helps
if not, I'll have to dig into it and see what the actual problem is
Jimmy K. Oak
@jimmyoak
Sep 15 2016 14:57
@mbogoevici @spencergibb Thank you very much!
@bitsofinfo hope you solved your doubts
Marius Bogoevici
@mbogoevici
Sep 15 2016 15:09
@jimmyoak no problem, thanks for reaching out!
Jimmy K. Oak
@jimmyoak
Sep 15 2016 15:12
@mbogoevici do you have an aprox release date of an stable version?
Marius Bogoevici
@mbogoevici
Sep 15 2016 15:13
RC2 should be stable enough … RELEASE is coming out next week, but we only have docs and one independent feature (improved Kerberos auth support)
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:14
Hello all! This is my first question here ever, so be gentle! I'm trying to get a Hystrix Fallback working in a @FeignClient. I'm on Brixton.SR5. I'm implemented it like here: spring-cloud/spring-cloud-netflix#762 , but without the intermediary Interface. When I run (keeping the service the Feign Client calls offline so it will fail, I get a HystrixRuntimeException, saying "[method name] failed and no fallback available.". Any idea what the problem might be?
Dave Syer
@dsyer
Sep 15 2016 15:22
You didn't provide a fallback?
(Not sure what you mean by "intermediary Interface")
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:24
In that example there is a "FooClientInterface" which "FeignFooClient" extends and "FooFallback" implements. Instead, I just had "FooFallback" implement "FeignFooClient"
Dave Syer
@dsyer
Sep 15 2016 15:25
One interface is enough
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:25
My version of "FooFallback" has the @Component annotation on the class as well.
Dave Syer
@dsyer
Sep 15 2016 15:25
Is the "FooFallback" a @Bean (or a @Component)?
i.e. is it managed by Spring?
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:26
@Component
Dave Syer
@dsyer
Sep 15 2016 15:27
Maybe it didn't get scanned?
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:29
I debugged the code and in the "invoke" method of "HystrixInvocationHandler" I can inspect the Method, get the class and find the FeignClient annotation on it. In there, the 'fallback' is set to "void"
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:29
what does your @FeignClient annotation look like?
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:30
I assumed it is like what you said @dsyer that it wasn't being scanned. But after adding the @Bean in my Application class, the same thing happened.
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:30
@FeignClient(
name = "my-service",
fallback = MyClientFallback.class
)
Thanks @ryanjbaxter. I looked at the code and it seems mine is identical setup wise
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:32

and u have tried adding

@Bean
public MyClientFallback myClientFallback() {
  return new MyClientFallback();
}

to your application?

can u provide the whole stacktrace?
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:33
Yeah, when I thought the package might not have been automatically scanned I did that in an effort to have it picked up, but it didn't help.
Dave Syer
@dsyer
Sep 15 2016 15:34
Maybe it's failing higher or lower in the stack?
The exception stack trace would help
Or a sample
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:35
trying to dig that up now
Okay, here is stack trace: http://pastebin.com/a97CVCCp
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:40
However, I think it isn't helpful. To trigger the fallback being used I disabled other services. The error seems directly related to that.
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:46
@ryanjbaxter : In your example the method that calls the feign client is annotated with @HystrixCommand. Is that required? I assume it isn't but that is the main difference I see.
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:47
no its not
Spencer Gibb
@spencergibb
Sep 15 2016 15:47
@matdelong_twitter it would be helpful to see your feign interface and fallback
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:52
Sure, I've added also the Application.json: http://pastebin.com/jTcG4uP1
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:53
i dont think its a problem with your fallback implementation not being found bc if that was the case your application would not even start because we would throw an exception when creating the FeignClient
bitsofinfo
@bitsofinfo
Sep 15 2016 15:56
is this supported?

    @Output("${my.prop.name}")
    public MessageChannel myOutboundChannel();
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:58
Hmmm, but I get that "... and no fallback available." error message @ryanjbaxter. And when I'm debugging, it doesn't get into my fallback's method. (And it might be a red-herring of inspecting during reflection, but the 'void' value at runtime for the fallback confuses me.)
Spencer Gibb
@spencergibb
Sep 15 2016 15:58
@matdelong_twitter I think at this point we probably need a sample project
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:59
what i am saying is if you remove the fallback bean and try to run the app it should fail to start
Mat DeLong
@matdelong_twitter
Sep 15 2016 15:59
Yeah, I'll throw one together for you. Thanks. I'm not sure if I hope my sample project reproduces the problem or doesn't.
Ryan Baxter
@ryanjbaxter
Sep 15 2016 15:59
also you shouldnt need the @EnableHystrix annotation, but that isnt causing the problem
Mat DeLong
@matdelong_twitter
Sep 15 2016 16:02
yeah, I added that out of desperation :) I'll come back tomorrow and hopefully bother you guys more with a sample project.
Thanks for the help
bitsofinfo
@bitsofinfo
Sep 15 2016 16:04
yeah.. I guess not, it blows up here KafkaMessageChannelBinder receives ${my.prop.name}
    public static void validateTopicName(String topicName) {
        try {
            byte[] utf8 = topicName.getBytes("UTF-8");
            for (byte b : utf8) {
                if (!((b >= 'a') && (b <= 'z') || (b >= 'A') && (b <= 'Z') || (b >= '0') && (b <= '9') || (b == '.')
                        || (b == '-') || (b == '_'))) {
                    throw new IllegalArgumentException(
                            "Topic name can only have ASCII alphanumerics, '.', '_' and '-'");
                }
            }
        }
        catch (UnsupportedEncodingException e) {
            throw new AssertionError(e); // Can't happen
        }
    }
Odín del Río
@odin-delrio
Sep 15 2016 16:17
@mbogoevici as @jimmyoak said, thank you so much for the support, we are already asking to our devops guys to upgrade our kafka version :grin:
Marius Bogoevici
@mbogoevici
Sep 15 2016 16:17
@odin-delrio very welcome
Spencer Gibb
@spencergibb
Sep 15 2016 16:39
@aabutaleb can you try setting turbine.combineHostPort=false?
David Welch
@dwelch2344
Sep 15 2016 18:54
hey, does SC + Feign support returning ResponseEntity? Or some other object with headers + status code bundled in?
Spencer Gibb
@spencergibb
Sep 15 2016 18:55
yup, ResponseEntity.
David Welch
@dwelch2344
Sep 15 2016 18:55
you guys just thought of everything :D
David Welch
@dwelch2344
Sep 15 2016 20:16
hmmm @spencergibb: seem to only get back a null. Any ideas why?
Spencer Gibb
@spencergibb
Sep 15 2016 20:16
nope