These are chat archives for spring-cloud/spring-cloud

23rd
May 2018
CH4:D
@chad_d_stud_twitter
May 23 2018 02:13
you can theoretically set the IP via cloud config
brought me to tears when one of our devs changed the behavior of the IP lookup without telling me
you can set default eureka ip address when the app starts then once it syncs to the cloud config server it pulls an updated one. do note that cloud config keys takes precedence over local keys
CH4:D
@chad_d_stud_twitter
May 23 2018 02:21
call the /actuator/refresh if you have new config changes. no need to rebuild or redeploy
Marcos Barbero
@marcosbarbero
May 23 2018 02:59
He mentioned he is not using cloud Config
CH4:D
@chad_d_stud_twitter
May 23 2018 03:53
if wants the behavior he wants, he SHOULD XD
vikrantch-hk
@vikrantch-hk
May 23 2018 04:50
@marcosbarbero I have tried almost everything I have found on google , but still eureka is taking 3 minutes to deregister clients which have been shut down ungracefully
CH4:D
@chad_d_stud_twitter
May 23 2018 04:54
yeah eureka has a flaky unregister algorithm
we have 7 instances of the same app running in our server
It won’t kill those instances
it made my team think that the changes we committed to our code pipeline was broken when in fact it was just eureka routing it to an older running instance
funny thing is it works fine once it’s running locally
vikrantch-hk
@vikrantch-hk
May 23 2018 05:23
@chad_d_stud_twitter did you get any solution
CH4:D
@chad_d_stud_twitter
May 23 2018 05:24
@vikrantch-hk someone’s looking into but it’s still an outstanding issue for us
Jose Armesto
@fiunchinho
May 23 2018 07:08
when using spring cloud, is ZoneAwareLoadBalancer the default LoadBalancer used?
Jose Armesto
@fiunchinho
May 23 2018 07:17
is there a way to debug why the loadbalancer is choosing the instances that is choosing?
Marcos Barbero
@marcosbarbero
May 23 2018 10:48
@vikrantch-hk that’s really odd, is it just a plan eureka-client config?
@fiunchinho as far as I remember that’s the one. By default it uses round robin algorithm to pick the instance.
Is there something wrong going on? Any unexpected behavior?
nhmarujo
@nhmarujo
May 23 2018 11:30
Hi everyone. Any news regarding RC2?
Jose Armesto
@fiunchinho
May 23 2018 11:31
so marcos, I have news!
setting the hostname as IP address fixed yesterday's issue, because clients started using the IP field from the info got from eureka
BUT I found that clients had hardcoded a ribbon property that forces to use the IP address instead of hostname
so I'm positive that was the reason I was getting requests to the IP instead of hostname
now, once that's fixed, I observe that requests coming to the application being migrated to kubernetes, they all end up on the new started instance running on kubernetes, besides I have 3 other instances running on my old AWS infrastructure
so no matter what, all traffic goes to the instance running on k8s, and if I stop it, then all traffic starts being balanced between the three AWS instances normally
Marcos Barbero
@marcosbarbero
May 23 2018 11:38
Are you configuring any zone affinity?
Can you share your eureka config?
Jose Armesto
@fiunchinho
May 23 2018 11:40
the app is using this
eureka:
  instance:
    hostname: ${INGRESS_DNS}
    ipAddress: ${INGRESS_DNS}
    nonSecurePort: 80
    preferIpAddress: false
    instanceId: ${POD_NAME}
the client
eureka:
  instance:
    preferIpAddress: false
feign:
  hystrix:
    enabled: true

hystrix.command.default.execution.isolation.strategy: SEMAPHORE
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 800

ribbon:
  ConnectTimeout: 1000
  ReadTimeout: 2000
  MaxAutoRetries: 1

myapp:
  value: "myapp"
  ReadTimeout: 1000
  ConnectTimeout: 1000
  ribbon:
    UseIPAddrForServer: true
the last line is the one causing the first problem that I believe it's fixed once I remove it, so I guess we can just ignore it
Marcos Barbero
@marcosbarbero
May 23 2018 11:43
Where are the zones configured? Or defaultZone?
Jose Armesto
@fiunchinho
May 23 2018 11:45
they both use this config, and we don't specify any AZ (we have 1 eureka instance on each eu-west-1a, eu-west-1b and eu-west-1c)
eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: ${CLOUD_EUREKA}
    eurekaServerPort: 8080
    eurekaServerURLContext: eureka-server/v2
    region: ${EC2_REGION}
Marcos Barbero
@marcosbarbero
May 23 2018 11:48
Can you check what you have in the actuator /env endpoint?
Jose Armesto
@fiunchinho
May 23 2018 11:51
sure, what are you interested in?
Marcos Barbero
@marcosbarbero
May 23 2018 11:53
In the eureka Config
Jose Armesto
@fiunchinho
May 23 2018 11:57
it should be the same I just pasted it, shouldn't it?
Marcos Barbero
@marcosbarbero
May 23 2018 11:58
I know, I just want to be sure :smile:
Jose Armesto
@fiunchinho
May 23 2018 11:59
applicationConfig: [classpath:/application-k8s.yml]: {
eureka.instance.hostname: "myapp.public-dns.com",
eureka.instance.ipAddress: "myapp.public-dns.com",
eureka.instance.nonSecurePort: 80,
eureka.instance.preferIpAddress: false,
eureka.instance.instanceId: "myapp-2752492436-jprtz"
}
that's the instance running on kubernetes
and it also appears the env vars for CLOUD_EUREKA and EC2_REGION under systemEnvironment
Marcos Barbero
@marcosbarbero
May 23 2018 12:02
Do you have the eureka.client properties ?
Jose Armesto
@fiunchinho
May 23 2018 12:02
and the client
applicationConfig: [classpath:/application-pre.yml]: {
eureka.client.registerWithEureka: true,
eureka.client.fetchRegistry: true,
eureka.client.useDnsForFetchingServiceUrls: true,
eureka.client.eurekaServerDNSName: "eureka.company.io",
eureka.client.eurekaServerPort: 8080,
eureka.client.eurekaServerURLContext: "eureka-server/v2",
eureka.client.region: "eu-west-1",
spring.metrics.export.enabled: false,
spring.sleuth.sampler.percentage: 1,
logging.config: "classpath:logback-aws.xml",
logback.access.config: "classpath:logback-access-aws.xml"
}
Marcos Barbero
@marcosbarbero
May 23 2018 12:11
All of them use the same eureka server?
Jose Armesto
@fiunchinho
May 23 2018 12:11
yes
we have 3 eureka instances on the same AWS region, one per AZ
everything worked fine, but as soon as an instance from kubernetes registers itself on eureka as instance of APPLICATION-A, then all requests to APPLICATION-a go to that instance
if I remove it, everything goes back to normal
my app instances on AWS all get requests
Marcos Barbero
@marcosbarbero
May 23 2018 12:17
That’s really odd
Your configuration seems legit
Jose Armesto
@fiunchinho
May 23 2018 12:18
ok this is interesting now: trying to find differences in Eureka info for the AWS instances and the k8s ones, the AWS instances have metadata for the data center info, but the k8s have datacenter "MyOwn" and its metadata is empty
but we don't when the app runs on k8s
vikrantch-hk
@vikrantch-hk
May 23 2018 12:26
@marcosbarbero eureka server is spring cloud netflix , client is without spring cloud
Marcos Barbero
@marcosbarbero
May 23 2018 12:30
@fiunchinho that’s correct, if you are running on AWS it has some specific information about the datacenter, what’s not present if you deploy it anywhere else
Jose Armesto
@fiunchinho
May 23 2018 12:31
but should it matter on the load balancing?
Marcos Barbero
@marcosbarbero
May 23 2018 12:31
I don’t know how much it affect the routing part, although I don’t believe it should
@vikrantch-hk I have used the same setup before and it worked.
Can you use spring cloud Netflix also in the clients or is it too difficult to implement right now?
Jose Armesto
@fiunchinho
May 23 2018 12:33
so any hints Marcos?
Marcos Barbero
@marcosbarbero
May 23 2018 12:35
At this moment there’s nothing that comes to my mind. I’m also doing a little research, let’s see what I can find
Jose Armesto
@fiunchinho
May 23 2018 12:36
mm....
adding a random availability zone in the metadata when registering on eureka, it seems to fix it? wtf?
applicationConfig: [classpath:/application-k8s.yml]: {
eureka.instance.hostname: "myapp.public-dns.com",
eureka.instance.ipAddress: "myapp.public-dns.com",
eureka.instance.nonSecurePort: 80,
eureka.instance.preferIpAddress: false,
eureka.instance.instanceId: "myapp-2752492436-jprtz",
eureka.instance.metadataMap.zone: "eu-west-1a"
}
I hardcoded the metadata there, and now it looks like it's working
but I can't understand why
vikrantch-hk
@vikrantch-hk
May 23 2018 12:39
@marcosbarbero it's not possible to use spring cloud on client as it's a legacy application, can you please share some sample where I can find properties which control eureka registry in case of non graceful client shutdown
Marcos Barbero
@marcosbarbero
May 23 2018 12:45
@fiunchinho that’s new for me! Maybe you should file an issue explaining everything? Maybe it’s just a matter of documentation
Jose Armesto
@fiunchinho
May 23 2018 12:45
maybe not that many people using eureka with clients inside and outside aws?
where should I create the issue?
Marcos Barbero
@marcosbarbero
May 23 2018 12:53
Create it in the spring cloud Netflix repository on github
I do believe that’s the case, not a lot of people are mixing AWS and non AWS environment
@vikrantch-hk I don’t have any by now.
What if you implement a retry policy for your requests?
vikrantch-hk
@vikrantch-hk
May 23 2018 12:58
@marcosbarbero is there some sample code to help
Marcos Barbero
@marcosbarbero
May 23 2018 13:04
Are you using Zuul to route the requests to eureka clients?
vikrantch-hk
@vikrantch-hk
May 23 2018 13:10
yes
Marcos Barbero
@marcosbarbero
May 23 2018 13:21
You can configure it to retry
If I’m not wrong it’s zuul.retryable=true
vikrantch-hk
@vikrantch-hk
May 23 2018 13:38
its true
vikrantch-hk
@vikrantch-hk
May 23 2018 13:53
@marcosbarbero it's deregistering now in almost 1 minute
but I am not sure which property is actually working
Marcos Barbero
@marcosbarbero
May 23 2018 13:55
I can tell you it won’t deregister immediately, it always takes time because zuul itself also keeps a cache
vikrantch-hk
@vikrantch-hk
May 23 2018 13:55
so does it shows client on eureka console also
Marcos Barbero
@marcosbarbero
May 23 2018 13:56
The retryable flag is just saying “if the request fails then do it again in the next node”
vikrantch-hk
@vikrantch-hk
May 23 2018 13:56
eureka:
  client:
    register-with-eureka: true
    fetch-registry: true
    serviceUrl:
      defaultZone: http://localhost:8080/eureka/
  instance:
    leaseRenewalIntervalInSeconds: 1
    leaseExpirationDurationInSeconds: 2
    lease-renewal-interval-in-seconds: 1
    lease-expiration-duration-in-seconds: 2
    preferIpAddress: true
server.port: 8080
third:
  party:
    api:
      access:
        user:
          name: api
        password: yegfhsgdegjkldcvgftykaqbfty
hystrix:
  command:
    api:
      execution:
        timeout:
          enabled: false
        isolation:
          thread:
            timeoutInMilliseconds: 60000
    hkSecurity:
      execution:
        timeout:
          enabled: false
        isolation:
          thread:
            timeoutInMilliseconds: 60000
eureka.server.responseCacheUpdateInvervalMs: 1000
eureka.server.registry-sync-retry-wait-ms: 500
eureka.server.a-sgcache-expiry-timeout-ms: 1000
eureka.server.evictionIntervalTimerInMs: 1000
eureka.server.renewalThresholdUpdateIntervalMs: 1000
eureka.server.peer-eureka-nodes-update-interval-ms: 3000
ntervalTimerInMseureka.server.renewal-threshold-update-interval-ms: 1000
eureka.eipBindRebindRetries: 0
aws.region.static: ap-south-1
eureka.isAws: true
eureka.datacenter: cloud
eureka.instance.metadataMap.zone: ap-south-1
eureka.client.region: ap-south-1
eureka.client.prefer-same-zone-eureka: true
eureka.client.availability-zones.ap-south-1: 'ap-south-1a,ap-south-1b'
eureka.client.serviceUrl.ap-south-1a: http://10.0.10.97:8080/eureka/
eureka.client.serviceUrl.ap-south-1b: http://10.0.20.97:8080/eureka/
Marcos Barbero
@marcosbarbero
May 23 2018 13:56
On eureka dashboard it should dismiss a lot faster
vikrantch-hk
@vikrantch-hk
May 23 2018 13:57
nope that is not happening on eureka console it is taking 1 minute
above is app-prod yaml
below is app yaml
spring:
  application:
    name: gateway-service
  profiles:
    active: prod
  cache:
    cache-names: appConfig
    caffeine:
      spec: expireAfterAccess=180s
  cloud:
    refresh:
      enabled: false
eureka.server.enableSelfPreservation: false
management:
  endpoint:
    health:
      enabled: true
ribbon:
  ConnectTimeout: 15000
  ReadTimeout: 15000
zuul:
  host:
    socket-timeout-millis: 60000
  #retryable: true
  sensitiveHeaders:
  routes:
    api:
      path: /api/**
      service-id: api
      strip-prefix: false
    hkSecurity:
      path: /hkSecurity/**
      service-id: hkSecurity
      strip-prefix: true
    feed:
      path: /feed/**
      service-id: api
      strip-prefix: false
feign.hystrix.enabled: true
hystrix.command.default.fallback.enabled: true
hystrix.command.default.circuitBreaker.enabled: true
earlier it was taking 3 minutes now it's taking 2 to disappear dead client on eureka dashboard
Marcos Barbero
@marcosbarbero
May 23 2018 13:59
You don’t need those eureka.server properties if your application isn’t the eureka server
If you check the /actuator/env endpoint are the properties the same as the Config file?
vikrantch-hk
@vikrantch-hk
May 23 2018 14:03
/actuator/env is not working
do I need to enable some config for actuator
Marcos Barbero
@marcosbarbero
May 23 2018 14:09
You do, it’s disabled by default
vikrantch-hk
@vikrantch-hk
May 23 2018 14:14
actuator dependency is there
do I need to configure some property also
Marcos Barbero
@marcosbarbero
May 23 2018 14:19
Yes, you do.
I don’t recall it right now, can you check the docs and find how to enable this endpoint?
vikrantch-hk
@vikrantch-hk
May 23 2018 14:19
sure
Jose Armesto
@fiunchinho
May 23 2018 15:41
hey Marcos, I wrote the issue spring-cloud/spring-cloud-netflix#2964 , any feedback?
Marcos Barbero
@marcosbarbero
May 23 2018 15:42
Seems great to me
Right now I can’t run any tests myself, I’ll try to take a look next week
Jose Armesto
@fiunchinho
May 23 2018 15:43
I'd really appreciate it if you could
Marcos Barbero
@marcosbarbero
May 23 2018 15:43
Maybe someone else will take a look before me
I know there’s an event going on in Barcelona this week, so I doubt it will be picked up by this week
Jose Armesto
@fiunchinho
May 23 2018 15:44
coincidentally, I live in Barcelona but I can't join :/
Marcos Barbero
@marcosbarbero
May 23 2018 15:48
Porque no amigo? :smile:
Jajaja just kidding
Jose Armesto
@fiunchinho
May 23 2018 15:49
I'm originally from Galicia, so we could talk portuguese more or less :P
Marcos Barbero
@marcosbarbero
May 23 2018 15:49
Nice haha