Hi to all
There is a zuul gateway with the following configuration in production mode:
http:
multipart:
max-file-size: 20MB
max-request-size: 20MB
application:
name: amad-gateway
ribbon:
ConnectTimeout: 20000000
ReadTimeout: 20000000
maxAutoRetries: 1
MaxAutoRetriesNextServer: 2
MaxTotalHttpConnections: 1600
MaxConnectionsPerHost: 800
eureka:
enabled: true
zuul:
hystrix:
command:
default:
execution:
isolation:
strategy: SEMAPHORE
semaphore:
maxConcurrentRequests: 100000
fallback:
isolation:
semaphore:
maxConcurrentRequests: 100000
semaphore:
maxSemaphores: 100000
But some times it crushes for about 3 minutes when the number of requests up
It is necessary to mention the the version of zuul in
Finchley.RELEASE
And the number of our users in the maximum level is 800
Where is wrong?
and something else configuration, I must do for improving the performance?
Can you upgrade to
Finchley.SR3 and see if the problem persists?
@marcingrzejszczak You mean it will solve my problem? and there wont be crushed time?
I don't understand your problem really but you should always upgrade to the latest version
@marcingrzejszczak When the number of users are increase or another hand when the number of request increase to gateway, it is crushed for 3 to 5 minutes
@marcingrzejszczak How do I tune the gateway or scale it?
@spencergibb is the gateway master you'll have to wait for him for the reply or someone from the community
@spencergibb Hi,
There is a zuul gateway with the following configuration in production mode:
There is a zuul gateway with the following configuration in production mode:
multipart:
max-file-size: 20MB
max-request-size: 20MB
application:
name: amad-gateway
ribbon:
ConnectTimeout: 20000000
ReadTimeout: 20000000
maxAutoRetries: 1
MaxAutoRetriesNextServer: 2
MaxTotalHttpConnections: 1600
MaxConnectionsPerHost: 800
eureka:
enabled: true
zuul:
hystrix:
command:
default:
execution:
isolation:
strategy: SEMAPHORE
semaphore:
maxConcurrentRequests: 100000
fallback:
isolation:
semaphore:
maxConcurrentRequests: 100000
semaphore:
maxSemaphores: 100000
But some times it crushes for about 3 minutes when the number of requests are increased
@spencergibb
It is necessary to mention the the version of zuul in
And the number of our users in the maximum level is 800
Where is wrong?
and something else configuration, I must do for improving the performance?
It is necessary to mention the the version of zuul in
Finchley.RELEASEAnd the number of our users in the maximum level is 800
Where is wrong?
and something else configuration, I must do for improving the performance?
@matin-reza AFAIK, Unless you use non blocking IO (sprign cloud gateway comes with netty which provides non blocking IO), there is no way you can reach
100000 of concurrent requests with semaphore isolation
MaxConnectionsPerHost: 800
With semaphore isolation, each downstream request is run in the calling thread (meaning the thread handling request in zuul is blocked till the downstream has responded)
Since we are tying up resources of thread for the whole request-response cycle in a blocking manner, you cannot get
100000 parallel simulatenous requests on any hardware
This is due to fact that each thread in java has a fixed overhead of about ~1-2MB
So, for
100000 simultaneous requests, you need a RAM to the tune of 100000 * 1MB => 100GB. This is just for holding thread stacks. If you add all the heap requirements, you are looking at a very large RAM space. Please note that I have not added the cost of context switch CPU needs to perform to schedule these threads on to the underlying cores
Overall, you can only achieve high throughput thru non blocking IO. Spring cloud gateway supports non blocking IO, due to which, threads are not reserved for the whole request response cycle. Rather they are reused across all parallel requests
@spencergibb Is there any other reason for this limit on the
MaxConnectionsPerHost?
TCP supports upto 60K+ (65K - 1K reserved) connections between a given source host + target host:port combo. Whats the reason for restricting this to just
800?
@matin-reza You will not be able to timeout your requests in semaphore mode unless the underlying TCP connection does a read timeout. So, using SEMAPHORE is not a great idea to begin with. @spencergibb Please correct me if I am wrong
@spencergibb Continuing from twitter
https://twitter.com/thekalinga/status/1102589484556869632
Is this how loadbalancing works in Spring Cloud Gateway
- Using
LoadBalancerClient, we select the server based on serviceId - Construct url based on this server & use regular netty client (assuming we are using defaults) to make non blocking outbound call?
And we skip over http client facilities of ribbon completely
spring cloud has not used the http client facilities of ribbon for years
Even in the older versions, spring cloud with zuul 1?
older versions of the release train, Angel maybe Brixton but they are long unsupported.
In
Greenwich.RELEASE, when I make a call using @LoadBalanced restTemplate, may I know who actually makes the underlying http request once the serviceId -> server resolution is done
This in the context of
zuul 1
zuul doesn't use rest template
Sorry.. I think I mixedup couple of thing here
Assuming we are using client side loadbalancer (ribbon) integrated with service registry (eureka), in Spring cloud
Greenwich.RELEASE- If I use
@LoadBalancedrestTemplate, who does the actual underlying call? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending? - If I use
@FeignClient, with who does the actual underlying call? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending? - Incase
zuul 1, who does the actual underlying upstream calls? Ribbon with its own http client (or) someone else. If someone else, appreciate if you can share the class responsible for sending?
ribbon never makes the underlying call. for 1 and 2 it uses the framework default (java http url I believe). 3 is apache http client
I might be mistaken about feign and it might use apache as the default as well
spring cloud gateway uses reactor netty httpclient
Oh Ok.. I was of the impression that
restTemplate delegates the whole request processing when using serviceId to LoadBalancerInterceptor, which further delegates everything to Ribbon, who finally makes the undlying call using its own htp client
initially it did years ago
Can u give me a clue on how the delgation works now
I will dig the further into the source to find more info
There is no delegation. Ribbon is plugged in to select an instance (host:port) and update the url. Then normal operations occur.
What do you want to know specifically?
I am trying to get my head around how spring internal APIs like
RestTemplate are integrated with Ribbon, Hystrix, Feign & Zuul
rest template is not integrated with hystrix feign or zuul
there is a ribbon interceptor that plugs into rest template
why do you want to know this?
I need to train others :) Tho I know how to use the APIs at the developer level, its better for me to explain in depth till the framework level on how everything is wired. I am trying to gain that level of knowledge
I see that there is a org.springframework.cloud.netflix.ribbon.RibbonHttpRequest#executeInternal, which does delegate to netflix client. Is this class not used during RestTemplate flow
no it is not be default
Is there a process by which I can follow the actual flow? (other than blind debugging)
not that I'm aware of
I have to follow the old school approach to understanding the flow by debugging then :)
Thanks a lot for your help!!
in all cases it is an interceptor of some kind.
True but I see the interceptor (in
RestTemplate case) is delegating the actual calls to Requests for some reason which is where I am quite confused
I see that RibbonLoadBalancerClient implements LoadBalancerClient& loadBalancer.execute(..) does this T returnVal = request.apply(serviceInstance); (delegates to LoadBalancerRequest)
So, the interceptor is delegating execution back to request internally. Because of these handshakes, its is difficult to know who is making the ultimate underlying call. Which is the source of confusion
right, but the request is created in
LoadBalancerInterceptor. The ribbon execute is just to choose the server and record some metrics. The request is a parameter.
Sure.. I see that. The only aspect thats confusing is the handshakes restTemplate -> interceptor -> request -> ....???... -> actual call
Is this understanding correct?
- Given
http://my-serviceas URL,org.springframework.cloud.client.loadbalancer.ServiceRequestWrapperconverts this virtual URL to real URL based onServiceInstance - My assumption is,
ClientHttpRequestExecutionmakes the ultimate http call (whichLoadBalancerRequestwraps) - So, when
RibbonLoadBalancerClient.executeis run, the load balancer request create atorg.springframework.cloud.client.loadbalancer.LoadBalancerRequestFactory.createRequestinvokesexecution.executeusing the uri created using ribbon
After this, I checked who implemented ClientHttpRequestExecution. I see only one implementation in the classpath i.e org.springframework.http.client.InterceptingClientHttpRequest.InterceptingRequestExecution
But he is creating another ClientHttpRequest. I could not follow why & for what. So the actual library that ultimately makes calls is a bit of a mystery
But that is Rest Template from framework, not spring cloud
You are right.. I'll have dig deep into Spring aswell to know a bit of internals :)
Thanks a lot for your help!
@spencergibb Whats the use of
RouteToRequestUrl filter in gateway?
The documentation says what it does, but am a bit confused about it
From Documentation
The RouteToRequestUrlFilter runs if there is a Route object in the ServerWebExchangeUtils.GATEWAY_ROUTE_ATTR exchange attribute. It creates a new URI, based off of the request URI, but updated with the URI attribute of the Route object. The new URI is placed in the ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR exchange attribute.
If the URI has a scheme prefix, such as lb:ws://serviceid, the lb scheme is stripped from the URI and placed in the ServerWebExchangeUtils.GATEWAY_SCHEME_PREFIX_ATTR for use later in the filter chain.
Is it mainly for supporting loadbalancing across non-http protocols?
nope, it gets the uri from the route matched and adds it to echange attribute
Since this info can be fetched from
ServerWebExchangeUtils.GATEWAY_ROUTE_ATTR directly, Appreciate if you can give a usecase for storing the URI under a seperate variable again
to allow other filters to modify it later
Plz forgive me for basic questions. Cant the first filter that need to modify it set the attribute in the exchange & pass it along? Why have a dedicated filter for doing
exchange.setAttribute(SHARED_KEY, exchange.getRequest().getURI())
If those that need it does it, we will have one less filter
I am sure I am missing something obvious. Appreciate if you can shed light on it