These are chat archives for spring-cloud/spring-cloud

6th
Sep 2016
nmquyet
@nmquyet
Sep 06 2016 10:32
I noticed that Spring Cloud Config Client keep locating environment from Config Server. Is this normal?
Located environment: name=application, profiles=[k8s], label=master, version=cc8f6b05b66efc3feace85f694ea6a3197f4883a
Ryan Baxter
@ryanjbaxter
Sep 06 2016 11:05
@stephlag for what?
Marcos Barbero
@marcosbarbero
Sep 06 2016 11:49
@nmquyet it's the configserver health check behavior.
nmquyet
@nmquyet
Sep 06 2016 12:13
Thank @marcosbarbero. It's good to know
Stephan L
@stephlag
Sep 06 2016 12:16

@ryanjbaxter for your message (late reply but I was on vacation!)

FYI jenkins has looked fine since merging the pull request, thanks @stephlag!

Ryan Baxter
@ryanjbaxter
Sep 06 2016 13:27
oh np ;)
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:15
@dsyer , I m seeing some stange behavior in eureka ..
when I restart servers in cloud foundry.. it registered the new instances and don't let go old instances
I know self preservation mode ..
another issue is design related... sometime my github goes down and if any app restart happens it failes to come up..as config server can't pull github proeprties
do we have any failover design for these kind of issues ?
Spencer Gibb
@spencergibb
Sep 06 2016 17:43
@pradeepkusingh there’s a spring one platform talk about customizing config server that talks about what one company did.
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:43
Which channel ?
Spencer Gibb
@spencergibb
Sep 06 2016 17:45
I don’t know if it is up yet
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:50
ok.
Thanks
Dave Syer
@dsyer
Sep 06 2016 17:52
Are you using spring-retry in the clients?
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:52
yes..
Dave Syer
@dsyer
Sep 06 2016 17:52
(That would help)
The server should be fairly robust as well. Maybe it depends on the nature of the outage.
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:53
unfortunately one day my github server was down for couple of hours.. and someone restarted the apps and it never came back.. caused big outage :(
so we are trying to explore more design which can help to avoid these kind of issues in future
Dave Syer
@dsyer
Sep 06 2016 17:54
The config servers all have local git clones
So they normally continue to operate if the server is down
Unless there is a bad bet outage and timeouts.
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:55
hmm , strange .. if Cloud foundry there is no File system .. so where it store ?
Dave Syer
@dsyer
Sep 06 2016 17:55
There's a file system
It's just not persistent
s/bet /net /
Spencer Gibb
@spencergibb
Sep 06 2016 17:56
IIRC Domo runs off of cloned repo’s with a cron that updates them, so they aren’t depenant on live git server.
pradeepkusingh
@pradeepkusingh
Sep 06 2016 17:56
great.. let me test.. it seems my understanding was wrong :)
Thanks
David Welch
@dwelch2344
Sep 06 2016 18:33

Ok @spencergibb, riddle me this: in our kubernetes cluster we have a 3-node consul setup. Working great for the most part. However, if we gracefully shut down one of the consul nodes, all services that initially pulled configs from that node start blowing up. It looks like ConfigWatch#watchConfigKeyValues() blows up ~81. The question is why?

I'm guessing the consulClient for the Watcher isn't happy w/ the new box / doesn't have a valid connection / etc. However, since all 3 consul nodes are behind a load balanced service, said consulClient should be able to use the same URL as before (and hit something healthy)

Stack trace generated is pretty straight forward. Looks like the KV response is straight up null (from the client lib). I know that's an external dependency, but is there / should there be a way to recover from this?

2016-09-02 00:13:53.748 [LEVEL=ERROR] [IID=dev-users-vhehz::1] [IUID=][ITENANT=][IURL=]          o.s.c.c.config.ConfigWatch               -      Error initializing listener for context config/users/

com.ecwid.consul.v1.OperationException: null
       at com.ecwid.consul.v1.kv.KeyValueConsulClient.getKVValues(KeyValueConsulClient.java:159) ~[consul-api-1.1.9.jar!/:na]
       at com.ecwid.consul.v1.kv.KeyValueConsulClient.getKVValues(KeyValueConsulClient.java:143) ~[consul-api-1.1.9.jar!/:na]
       at com.ecwid.consul.v1.ConsulClient.getKVValues(ConsulClient.java:399) ~[consul-api-1.1.9.jar!/:na]
       at org.springframework.cloud.consul.config.ConfigWatch.watchConfigKeyValues(ConfigWatch.java:81) ~[spring-cloud-consul-config-1.0.0.RELEASE.jar!/:1.0.0.RELEASE]
       at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) ~[na:na]
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_74]
       at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74]
       at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) [spring-context-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
       at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_74]
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_74]
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_74]
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_74]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
       at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
(sorry, repost from the long weekend)
Spencer Gibb
@spencergibb
Sep 06 2016 18:34
yeah, not sure. I’ve never run consul without an agent. Are you using agents or connecting directly to servers?
David Welch
@dwelch2344
Sep 06 2016 18:35
Hmmm good question. We're just using the spring-cloud-consul framework, so I've always thought it was directly to the server
Yeah, we're not running the agent in our pod. Sorry, coming off brutal weekend :P
Spencer Gibb
@spencergibb
Sep 06 2016 18:36
should still use agents AFAIK
The agents know about all of the servers
David Welch
@dwelch2344
Sep 06 2016 18:37
Maybe I'm misunderstanding...
oh, right. the agent API's on consul?
vs the catalog stuff?
Spencer Gibb
@spencergibb
Sep 06 2016 18:38
the consul binary runs in two modes, server and agent. You should have a cluster of ‘server’s (3 or 5 etc). And each node (or docker container) should have an agent connected to the servers.
s-c-consul should connect to the local agent.
David Welch
@dwelch2344
Sep 06 2016 18:39
... this is one of those moments where I feel like an absolute idiot
so I'd run the agent locally next to our s-c app and connect to it? and then it'd keep sync'd with the servers?
Spencer Gibb
@spencergibb
Sep 06 2016 18:41
yes
David Welch
@dwelch2344
Sep 06 2016 18:45
so I just sheepishly ran that scenario / setup by our ops guy, and we both sat in silence before finally saying "oh...". Hadn't even considered that setup, so we're putting it in play now. Thanks
and for the record: hitting the servers directly works great! just as long as the servers never die ;)
Spencer Gibb
@spencergibb
Sep 06 2016 18:57
:-) in communication between some of our users and hashicorp engineers that is the recommended way to run and the only way to scale for the kv store
David Welch
@dwelch2344
Sep 06 2016 19:53
yeah, totally makes sense. Looks like it's solving our woes (and required 0 code changes to boot)
So, 3 of us have been over the docs on Consul and done a number of setups / configs. None of us ever fully grasped the whole Agent <-> Catalog relationship until now. Probably something that a diagram could help with
(and def one of the areas we're going to add to our future presentations)
Spencer Gibb
@spencergibb
Sep 06 2016 20:17
yeah, I have a diagram in my consul presentation from S1P
Andrew Fitzgerald
@fitzoh
Sep 06 2016 20:59
Got a spring-cloud-consul question before I open an issue
If you use an @LoadBalanced rest template with spring cloud consul and use a serviceId, it doesn’t respect the query tag… Is this an old/fixed thing, expected behavior, or should be added as an issue?
Spencer Gibb
@spencergibb
Sep 06 2016 22:55
the ‘query tag’?
@Fitzoh :arrow_up:
Can you give me an example?
Andrew Fitzgerald
@fitzoh
Sep 06 2016 23:15
Sorry, service tag
Let’s say that 4 instances of a fooService are registered in consul
two of them have a tag of “a”, two of them have a tag of “b"
and the property spring.cloud.consul.discovery.serverListQueryTags.fooService=a
If you’re proxying through zuul, you’ll always end up on one of the two services tagged “a"
but if you use an @LoadBalanced restTemplate, you’ll end up hitting any of the four services if you make a request using the serviceId as the url ie restTemplate.getForObject(“fooService/thing”)
Andrew Fitzgerald
@fitzoh
Sep 06 2016 23:26
@spencergibb ^
Spencer Gibb
@spencergibb
Sep 06 2016 23:40
shouldn’t be that way, file an issue with steps to reproduce