These are chat archives for spring-cloud/spring-cloud

4th
Mar 2019
matin-reza
@matin-reza
Mar 04 08:45
Hi to all
There is a zuul gateway with the following configuration in production mode:
  http:
    multipart:
      max-file-size: 20MB
      max-request-size: 20MB
  application:
    name: amad-gateway
ribbon:
  ConnectTimeout: 20000000
  ReadTimeout: 20000000
  maxAutoRetries: 1
  MaxAutoRetriesNextServer: 2
  MaxTotalHttpConnections: 1600
  MaxConnectionsPerHost: 800
  eureka:
    enabled: true
zuul:
  hystrix:
    command:
      default:
        execution:
          isolation:
            strategy: SEMAPHORE
            semaphore:
              maxConcurrentRequests: 100000

        fallback:
          isolation:
            semaphore:
              maxConcurrentRequests: 100000
  semaphore:
    maxSemaphores: 100000
But some times it crushes for about 3 minutes when the number of requests up
matin-reza
@matin-reza
Mar 04 08:53
It is necessary to mention the the version of zuul in Finchley.RELEASE
And the number of our users in the maximum level is 800
Where is wrong?
and something else configuration, I must do for improving the performance?
Marcin Grzejszczak
@marcingrzejszczak
Mar 04 09:51
Can you upgrade to Finchley.SR3 and see if the problem persists?
matin-reza
@matin-reza
Mar 04 09:54
@marcingrzejszczak You mean it will solve my problem? and there wont be crushed time?
Marcin Grzejszczak
@marcingrzejszczak
Mar 04 09:54
I don't understand your problem really but you should always upgrade to the latest version
matin-reza
@matin-reza
Mar 04 09:56
@marcingrzejszczak When the number of users are increase or another hand when the number of request increase to gateway, it is crushed for 3 to 5 minutes
@marcingrzejszczak How do I tune the gateway or scale it?
Marcin Grzejszczak
@marcingrzejszczak
Mar 04 09:58
@spencergibb is the gateway master you'll have to wait for him for the reply or someone from the community
matin-reza
@matin-reza
Mar 04 10:01
@spencergibb Hi,
There is a zuul gateway with the following configuration in production mode:
    multipart:
      max-file-size: 20MB
      max-request-size: 20MB
  application:
    name: amad-gateway
ribbon:
  ConnectTimeout: 20000000
  ReadTimeout: 20000000
  maxAutoRetries: 1
  MaxAutoRetriesNextServer: 2
  MaxTotalHttpConnections: 1600
  MaxConnectionsPerHost: 800
  eureka:
    enabled: true
zuul:
  hystrix:
    command:
      default:
        execution:
          isolation:
            strategy: SEMAPHORE
            semaphore:
              maxConcurrentRequests: 100000

        fallback:
          isolation:
            semaphore:
              maxConcurrentRequests: 100000
  semaphore:
    maxSemaphores: 100000
But some times it crushes for about 3 minutes when the number of requests are increased
@spencergibb
It is necessary to mention the the version of zuul in Finchley.RELEASE
And the number of our users in the maximum level is 800
Where is wrong?
and something else configuration, I must do for improving the performance?
Ashok Koyi
@thekalinga
Mar 04 16:45
@matin-reza AFAIK, Unless you use non blocking IO (sprign cloud gateway comes with netty which provides non blocking IO), there is no way you can reach 100000 of concurrent requests with semaphore isolation
Spencer Gibb
@spencergibb
Mar 04 16:46
MaxConnectionsPerHost: 800
Ashok Koyi
@thekalinga
Mar 04 16:47
With semaphore isolation, each downstream request is run in the calling thread (meaning the thread handling request in zuul is blocked till the downstream has responded)
Since we are tying up resources of thread for the whole request-response cycle in a blocking manner, you cannot get 100000 parallel simulatenous requests on any hardware
This is due to fact that each thread in java has a fixed overhead of about ~1-2MB
So, for 100000 simultaneous requests, you need a RAM to the tune of 100000 * 1MB => 100GB. This is just for holding thread stacks. If you add all the heap requirements, you are looking at a very large RAM space. Please note that I have not added the cost of context switch CPU needs to perform to schedule these threads on to the underlying cores
Overall, you can only achieve high throughput thru non blocking IO. Spring cloud gateway supports non blocking IO, due to which, threads are not reserved for the whole request response cycle. Rather they are reused across all parallel requests
Ashok Koyi
@thekalinga
Mar 04 16:53
@spencergibb Is there any other reason for this limit on the MaxConnectionsPerHost?
TCP supports upto 60K+ (65K - 1K reserved) connections between a given source host + target host:port combo. Whats the reason for restricting this to just 800?
@matin-reza You will not be able to timeout your requests in semaphore mode unless the underlying TCP connection does a read timeout. So, using SEMAPHORE is not a great idea to begin with. @spencergibb Please correct me if I am wrong
Ashok Koyi
@thekalinga
Mar 04 17:01

@spencergibb Continuing from twitter

https://twitter.com/thekalinga/status/1102589484556869632

Is this how loadbalancing works in Spring Cloud Gateway

  1. Using LoadBalancerClient, we select the server based on serviceId
  2. Construct url based on this server & use regular netty client (assuming we are using defaults) to make non blocking outbound call?
And we skip over http client facilities of ribbon completely
Spencer Gibb
@spencergibb
Mar 04 17:02
spring cloud has not used the http client facilities of ribbon for years
Ashok Koyi
@thekalinga
Mar 04 17:02
Even in the older versions, spring cloud with zuul 1?
Spencer Gibb
@spencergibb
Mar 04 17:04
older versions of the release train, Angel maybe Brixton but they are long unsupported.
Ashok Koyi
@thekalinga
Mar 04 17:08
In Greenwich.RELEASE, when I make a call using @LoadBalanced restTemplate, may I know who actually makes the underlying http request once the serviceId -> server resolution is done
This in the context of zuul 1
Spencer Gibb
@spencergibb
Mar 04 17:08
zuul doesn't use rest template
Ashok Koyi
@thekalinga
Mar 04 17:10
Sorry.. I think I mixedup couple of thing here
Ashok Koyi
@thekalinga
Mar 04 17:15
Assuming we are using client side loadbalancer (ribbon) integrated with service registry (eureka), in Spring cloud Greenwich.RELEASE
  1. If I use @LoadBalanced restTemplate, who does the actual underlying call? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending?
  2. If I use @FeignClient, with who does the actual underlying call? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending?
  3. Incase zuul 1, who does the actual underlying upstream calls? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending?
Spencer Gibb
@spencergibb
Mar 04 17:17
ribbon never makes the underlying call. for 1 and 2 it uses the framework default (java http url I believe). 3 is apache http client
I might be mistaken about feign and it might use apache as the default as well
spring cloud gateway uses reactor netty httpclient
Ashok Koyi
@thekalinga
Mar 04 17:21
Oh Ok.. I was of the impression that restTemplate delegates the whole request processing when using serviceId to LoadBalancerInterceptor, which further delegates everything to Ribbon, who finally makes the undlying call using its own htp client
Spencer Gibb
@spencergibb
Mar 04 17:23
initially it did years ago
Ashok Koyi
@thekalinga
Mar 04 17:24
Can u give me a clue on how the delgation works now
I will dig the further into the source to find more info
Spencer Gibb
@spencergibb
Mar 04 17:31
There is no delegation. Ribbon is plugged in to select an instance (host:port) and update the url. Then normal operations occur.
What do you want to know specifically?
Ashok Koyi
@thekalinga
Mar 04 17:32
I am trying to get my head around how spring internal APIs like RestTemplate are integrated with Ribbon, Hystrix, Feign & Zuul
Spencer Gibb
@spencergibb
Mar 04 17:32
rest template is not integrated with hystrix feign or zuul
there is a ribbon interceptor that plugs into rest template
why do you want to know this?
Ashok Koyi
@thekalinga
Mar 04 17:38

I need to train others :) Tho I know how to use the APIs at the developer level, its better for me to explain in depth till the framework level on how everything is wired. I am trying to gain that level of knowledge

I see that there is a org.springframework.cloud.netflix.ribbon.RibbonHttpRequest#executeInternal, which does delegate to netflix client. Is this class not used during RestTemplate flow

Spencer Gibb
@spencergibb
Mar 04 17:39
no it is not be default
Ashok Koyi
@thekalinga
Mar 04 17:42
Is there a process by which I can follow the actual flow? (other than blind debugging)
Spencer Gibb
@spencergibb
Mar 04 17:44
not that I'm aware of
Ashok Koyi
@thekalinga
Mar 04 17:45

I have to follow the old school approach to understanding the flow by debugging then :)

Thanks a lot for your help!!

Spencer Gibb
@spencergibb
Mar 04 17:46
in all cases it is an interceptor of some kind.
Ashok Koyi
@thekalinga
Mar 04 17:47
True but I see the interceptor (in RestTemplate case) is delegating the actual calls to Requests for some reason which is where I am quite confused
Ashok Koyi
@thekalinga
Mar 04 17:54

I see that RibbonLoadBalancerClient implements LoadBalancerClient& loadBalancer.execute(..) does this T returnVal = request.apply(serviceInstance); (delegates to LoadBalancerRequest)

https://github.com/spring-cloud/spring-cloud-netflix/blob/435e94c3f58cc157a75f8bd4551b986454d7f965/spring-cloud-netflix-ribbon/src/main/java/org/springframework/cloud/netflix/ribbon/RibbonLoadBalancerClient.java#L144

So, the interceptor is delegating execution back to request internally. Because of these handshakes, its is difficult to know who is making the ultimate underlying call. Which is the source of confusion
Spencer Gibb
@spencergibb
Mar 04 17:58
right, but the request is created in LoadBalancerInterceptor. The ribbon execute is just to choose the server and record some metrics. The request is a parameter.
Ashok Koyi
@thekalinga
Mar 04 18:25

Sure.. I see that. The only aspect thats confusing is the handshakes restTemplate -> interceptor -> request -> ....???... -> actual call

Is this understanding correct?

  1. Given http://my-service as URL, org.springframework.cloud.client.loadbalancer.ServiceRequestWrapper converts this virtual URL to real URL based on ServiceInstance
  2. My assumption is, ClientHttpRequestExecution makes the ultimate http call (which LoadBalancerRequest wraps)
  3. So, when RibbonLoadBalancerClient.execute is run, the load balancer request create at org.springframework.cloud.client.loadbalancer.LoadBalancerRequestFactory.createRequest invokes execution.execute using the uri created using ribbon

After this, I checked who implemented ClientHttpRequestExecution. I see only one implementation in the classpath i.e org.springframework.http.client.InterceptingClientHttpRequest.InterceptingRequestExecution

But he is creating another ClientHttpRequest. I could not follow why & for what. So the actual library that ultimately makes calls is a bit of a mystery

Spencer Gibb
@spencergibb
Mar 04 18:50
But that is Rest Template from framework, not spring cloud
Ashok Koyi
@thekalinga
Mar 04 19:16
You are right.. I'll have dig deep into Spring aswell to know a bit of internals :)
Thanks a lot for your help!
Ashok Koyi
@thekalinga
Mar 04 20:13
@spencergibb Whats the use of RouteToRequestUrl filter in gateway?
The documentation says what it does, but am a bit confused about it

From Documentation


The RouteToRequestUrlFilter runs if there is a Route object in the ServerWebExchangeUtils.GATEWAY_ROUTE_ATTR exchange attribute. It creates a new URI, based off of the request URI, but updated with the URI attribute of the Route object. The new URI is placed in the ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR exchange attribute.

If the URI has a scheme prefix, such as lb:ws://serviceid, the lb scheme is stripped from the URI and placed in the ServerWebExchangeUtils.GATEWAY_SCHEME_PREFIX_ATTR for use later in the filter chain.


Is it mainly for supporting loadbalancing across non-http protocols?

Spencer Gibb
@spencergibb
Mar 04 21:56
nope, it gets the uri from the route matched and adds it to echange attribute
Ashok Koyi
@thekalinga
Mar 04 22:56
Since this info can be fetched from ServerWebExchangeUtils.GATEWAY_ROUTE_ATTR directly, Appreciate if you can give a usecase for storing the URI under a seperate variable again
Spencer Gibb
@spencergibb
Mar 04 22:57
to allow other filters to modify it later
Ashok Koyi
@thekalinga
Mar 04 23:01
Plz forgive me for basic questions. Cant the first filter that need to modify it set the attribute in the exchange & pass it along? Why have a dedicated filter for doing exchange.setAttribute(SHARED_KEY, exchange.getRequest().getURI())
If those that need it does it, we will have one less filter
I am sure I am missing something obvious. Appreciate if you can shed light on it