These are chat archives for spring-cloud/spring-cloud

3rd
Jun 2016
Thibaud Lepretre
@kakawait
Jun 03 2016 08:19
Do you think it could be a good idea to add SPeL on @HystrixCommand parameter like commandKey as same as http://docs.spring.io/spring/docs/current/spring-framework-reference/html/cache.html annotation? Because commandKey is not always as simple as you thought, I really want to generate it depending on method parameters! I know @HystrixCommand is not Spring classes but I was thinking about something similar as cache: a circuit breaker abstraction
SPeL or method generator (something a bit more dynamic)
Dave Syer
@dsyer
Jun 03 2016 08:22
The analogy with @Cacheable is pretty accurate
So we would need a new annotation, I guess
Thibaud Lepretre
@kakawait
Jun 03 2016 08:24
As same you started created @EnableCircuitBreaker (I think for handling other circuit breaker implementation), we need to create a global abstraction for circuit breaking
Dave Syer
@dsyer
Jun 03 2016 08:25
Yes. There are issues in github for that
Thibaud Lepretre
@kakawait
Jun 03 2016 08:25
Or opening a Netflix issue on Hystrix about providing at least a key generator (method or global class)
Dave Syer
@dsyer
Jun 03 2016 08:25
No work in progress except some experiments I did in Spring Retry
We need our own annotation at some point anyway. But you can choose.
Thibaud Lepretre
@kakawait
Jun 03 2016 08:26
Ok I will try to find it, for now I will use plain old HystrixCommand<T> class.
Dave Syer
@dsyer
Jun 03 2016 08:26
If you can send some code it won't be in 1.1 but we want to get 1.2 out in the summer.
Thibaud Lepretre
@kakawait
Jun 03 2016 08:28
Yeah I think 1.1 is not feasible but why not 1.2, at first we need to specify abstraction
@dsyer issue you talk is spring-cloud/spring-cloud-commons#21 ? Or there is another
Dave Syer
@dsyer
Jun 03 2016 08:30
That would be it
There might be others. But that's adequate.
Thibaud Lepretre
@kakawait
Jun 03 2016 08:30
Yeah I won't open a new I still prefer contribute on it
thank for your time
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:04
Hi there!
I'm having weird issues with the metrics stuff in a zuul based reverse proxy server. Sometimes we get timeouts in the log like "org.springframework.boot.actuate.autoconfigure.MetricsFilter;[Unable to submit counter metric 'status.500.userservice-rest.star-star']"
It uses the SimpleInMemoryRepository written by Dave
Caused by: javax.ejb.ConcurrentAccessTimeoutException: JBAS014373: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation@268c13ec - could not obtain lock within 5000MILLISECONDS
Dave Syer
@dsyer
Jun 03 2016 09:07
Sounds like a problem with the app
Is it under load?
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:07
seems to be the root cause for the exception, but 5 Sec is quite a long time, I don't understand why the lock wasn't removed :-(
it's a test server, so it's not that much load compared to the live server
Dave Syer
@dsyer
Jun 03 2016 09:07
Can you reproduce it?
Also, paste the stack trace somewhere please.
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:08
it doesn't happen often. Once per day at around 2am where there should be hardly any load
Maybe the metrics service should catch the exception. loosing metric points is not nice, but also not critical, I think
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:17
The service that has been called returned 500s each time that happened. Maybe there is a problem in the error handling for the forwarded calls
Dave Syer
@dsyer
Jun 03 2016 09:18
I really don't know. You have all the data. All I can see is that error message.
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:21
Hmm, I can't see much more either :-( (The machine is in a datacenter and I have only limited access via splunk...)
Dave Syer
@dsyer
Jun 03 2016 09:22
No stack trace?
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:22
Only the stacktrace that I copied to the pastebin above
Dave Syer
@dsyer
Jun 03 2016 09:23
Oops, missed it. Thanks.
It looks a bit garbled
Are there 2 exceptions at the same time?
It's only at WARN level as well, so I guess you already have you wish to ignore it and continue?
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:30
Hmm, there are two services that write to that log file (the reverse proxy and the "real" service) I was under the impression that log statements end up as a whole in the file, but you are right it looks like both are mixed
that is bad :-/
Dave Syer
@dsyer
Jun 03 2016 09:31
There is no concurrency and no background thread in the metric repository. So unless JBoss is being very sneaky with it, I don't understand how it can be related to the timeout.
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:32
Thats what I didn't understand too
Dave Syer
@dsyer
Jun 03 2016 09:32
If you can unmix the stack trace it might be clearer.
Is the "real" service using JBoss container threads?
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:33
I guess it's really just a mixup of log statements. Yes it's an ancient jboss where former employees thought it would be a good idea to patch cxf, so we can't update it.
That's why we introduced the reverse proxy so we can update the jboss and alter the behaviour that lead to the patched lib without breaking the clients until we moved all of them from soap to rest...
Dave Syer
@dsyer
Jun 03 2016 09:34
Sounds like an interesting use case
The exception in the metric repository looks like maybe a bug
You could raise a new ticket in Spring Boot
But I don't actually know how it happens
There's a NPE by the looks of it in a synchronized() block
Patrick Cornelißen
@pcornelissen
Jun 03 2016 09:38
yes, usually soap rpc doesn't allow null return values, but they wanted to have this, so they patched the Lib instead of changing the return values to something standard compliant. Now I have to rewrite the answers to some kind of null object, detect this in the reverse proxy and rewrite the response to the broken version. But at least we can then update the old jboss again ;-)
I'll create a spring boot issue later, thanks dave!
Patrick Cornelißen
@pcornelissen
Jun 03 2016 11:08
http://pastebin.com/J9tHjDPw is a clean stacktrace
Dave Syer
@dsyer
Jun 03 2016 12:03
Yes. Can you paste that into a new issue in Spring Boot please?
It's still only a warning.
Your app hasn't fallen over at that point
It's just under GC pressure
Paul
@asm0dey
Jun 03 2016 12:04
Hey guys! One of our services returns 404 errors (and that's normal behaviour because it's just routine searches in DB), and Hystrix interprets that as failures and breaks circuit. How to fix that?
Dave Syer
@dsyer
Jun 03 2016 12:05
Hystrix has no knowledge of HTTP codes
You must be throwing an exception from your HTTP client
Paul
@asm0dey
Jun 03 2016 12:05
We're using Feign
Dave Syer
@dsyer
Jun 03 2016 12:05
I guess you need to find a way to make the HTTP client not throw an exception then
What client are you using?
Paul
@asm0dey
Jun 03 2016 12:06
 private Response(int status, String reason, Map<String, Collection<String>> headers, Body body) {
    checkState(status >= 200, "Invalid status code: %s", status);
    this.status = status;
    this.reason = checkNotNull(reason, "reason");
    LinkedHashMap<String, Collection<String>>
        copyOf =
        new LinkedHashMap<String, Collection<String>>();
    copyOf.putAll(checkNotNull(headers, "headers"));
    this.headers = Collections.unmodifiableMap(copyOf);
    this.body = body; //nullable
  }
Dave Syer
@dsyer
Jun 03 2016 12:06
Where is that code?
Paul
@asm0dey
Jun 03 2016 12:06
In Response class
Dave Syer
@dsyer
Jun 03 2016 12:07
In Feign?
Paul
@asm0dey
Jun 03 2016 12:07
yup
Dave Syer
@dsyer
Jun 03 2016 12:07
I guess that's an issue for Feign then
not for Spring Cloud per se
Dave Syer
@dsyer
Jun 03 2016 12:08
If I were you I'd ask there (e.g. open an issue, send a PR)
In the meantime you can re-write your code to not use Feign I guess
Paul
@asm0dey
Jun 03 2016 12:10
Well, we're extensively using it: client code generation and so on :(
Dave Syer
@dsyer
Jun 03 2016 12:12
This line checks for >=400 but doesn't prevent the exception: https://github.com/Netflix/feign/blob/master/core/src/main/java/feign/Client.java#L173
I guess you can maybe configure hystrix to not treat those exceptions as errors
Paul
@asm0dey
Jun 03 2016 12:12
Yeah, sorry, first sample was incorrect
Dave Syer
@dsyer
Jun 03 2016 12:12
But it's an IllegalStateException
so that will be tricky
Paul
@asm0dey
Jun 03 2016 12:13
How can I configure it?
Dave Syer
@dsyer
Jun 03 2016 12:13
Hystrix has a wiki
If there's nothing there then you would need to dig into the source code for feign-hystrix
Paul
@asm0dey
Jun 03 2016 12:13
And 500s will be ignored too then, right?
Dave Syer
@dsyer
Jun 03 2016 12:14
Ignored?
Paul
@asm0dey
Jun 03 2016 12:14
Dave Syer
@dsyer
Jun 03 2016 12:14
500 is not special in that code
I don't know what you mean
Ah, you can't tell the difference.
True. Unless you look at the exception message.
Ugly
Seems like a reasonable thing to want though
Paul
@asm0dey
Jun 03 2016 12:15
Well, I do want hystrix to break circuit on 500-599 errors, but don't want circuit to be broken on 400-499
Sorry, my english is far from perfect, I know
Dave Syer
@dsyer
Jun 03 2016 12:18
Not bad at all, really
I know what you want.
It doesn't look like Feign supports it
But you should ask the developers of Feign
(some of them are here, but this isn't the right forum)
Paul
@asm0dey
Jun 03 2016 12:21
Of course I could make pull request, but I think that they'll tell me that I should implement ErrorDecoder. But problem is Error decoder is invoked only after hystrix :(
Andreas Evers
@andreasevers
Jun 03 2016 13:33
Hi, I’m trying to override a bean in the client-specific ApplicationContexts (instantiated through the SpringClientFactory), and I’m wondering if annotating my overriding configuration with @ConditionalOnBean(RibbonClientConfiguration.class) actually does the trick here?
Dave Syer
@dsyer
Jun 03 2016 13:48
@ConditionalOnBean is very sensitive to order of declaration
Andreas Evers
@andreasevers
Jun 03 2016 13:48
Dave Syer
@dsyer
Jun 03 2016 13:48
It's hardly ever useful except in auto config
Andreas Evers
@andreasevers
Jun 03 2016 13:49
How would you get a bean loaded into the client-specific ApplicationContexts while not necessarily in the main gateway context?
Dave Syer
@dsyer
Jun 03 2016 13:49
Unless you control the order of all the sources in the context
I suppose in this case you might be able to
The client specific contexts have sources specified in @*RibbonClient*
IIRC you can override them al if necessary
Maybe with a subclass?
Andreas Evers
@andreasevers
Jun 03 2016 13:52
Hmmm...
Andreas Evers
@andreasevers
Jun 03 2016 14:41
Can I actually use the @RibbonClient myself? I don’t want to create my own ribbonClients, I want to rely on the automatic generation of ribbonclients for each of my upstream microservices. Ideally I’d override a bean inside the RibbonClientConfiguration but I don’t think I can do that from inside my own applicationContext since a separate applicationContext is created for each service.
Seems similar to spring-cloud/spring-cloud-netflix#931 and spring-cloud/spring-cloud-netflix#935 but can’t exactly figure out how to make sense of it.
Dave Syer
@dsyer
Jun 03 2016 14:44
You're writing a library, not an app?
Andreas Evers
@andreasevers
Jun 03 2016 14:45
Just an app, a gateway installation
Could be I’m misunderstanding a part of the spring cloud & ribbon integration
Dave Syer
@dsyer
Jun 03 2016 14:57
Then you can just declare the context class in your @RibbonClient can't you?
Andreas Evers
@andreasevers
Jun 03 2016 15:06
I guess the root of my questions is this: can I declare beans and have them loaded by all ribbonClients but not the main applicationContext? Using @RibbonClient I’m specifying a specific ribbonClient, so that’s not really helpful as I want to load the beans for all the ribbonClients, and having @ComponentScan on those beans will load them into the main and the ribbonClient configurations…
Dave Syer
@dsyer
Jun 03 2016 15:13
@RibbonClients has a defaultConfiguration doesn't it?
Andreas Evers
@andreasevers
Jun 03 2016 15:16
Aha I’ll try that. Didn’t know that :+1: