These are chat archives for spring-cloud/spring-cloud

12th
Jan 2016
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 04:14

Hi, I have a microservice behind an API gateway (using Zuul) that I use to upload photos / files. The microservice acts as a bridge to a S3 storage. My problem is that the file is successfully uploaded to S3 and there are no errors on the microservice, but Zuul responded with a 500 error which is quite confusing.

This is Zuul's log: https://gist.github.com/wmfairuz/39854ee6f8a8dfddea61

I've already have these properties on Zuul:

ribbon:
  read-timeout: 600000

hystrix:
  command:
    default:
      execution:
        timeout:
          enabled: false

Thanks for any hints

hacbq
@hacbq
Jan 12 2016 04:56
@wmfairuz you can try to config Zuul. I get same bug with u :smile:
zuul:
  host:
    connect-timeout-millis: 60000
    socket-timeout-millis: 60000
ribbon:
  ReadTimeout: 60000
  ConnectTimeout: 60000
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 05:04
Is there any way to increase the timeout or disable the timeout only for a specific service?
hacbq
@hacbq
Jan 12 2016 05:07
This is timeout for zuul send to client. In fact, microservice still running :)
I dont know config for spectific service.
My s3 often need < 10s.
If large file, you should use a jobs
Daco
@dacofr
Jan 12 2016 06:00
@spencergibb ok, i'll try
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 07:03
@hacbq thanks
hm after some reading, from my understanding, Zuul will download the whole request body before proxy it to other services. In my case where I have client -> Zuul -> UploadService -> S3 storage, the uploaded file will be first uploaded to Zuul, then to my Upload Service, and then finally will be uploaded to S3.
Is my assumption correct?
Marcin Grzejszczak
@marcingrzejszczak
Jan 12 2016 07:07
@wmfairuz @hacbq - maybe this will be useful to you - http://projects.spring.io/spring-cloud/spring-cloud.html#_uploading_files_through_zuul ?
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 07:10
@marcingrzejszczak thanks! I knew I read about this some time ago and cannot re-find it this time.
Btw, does hystrix support per service timeout configuration? Like ribbon does with myService.ribbon.ReadTimeout?
Marcin Grzejszczak
@marcingrzejszczak
Jan 12 2016 07:11
np - ping us back if it's working again
Marcin Grzejszczak
@marcingrzejszczak
Jan 12 2016 07:24
@wmfairuz check out this issue spring-cloud/spring-cloud-netflix#329
if you provide the service id instead of the default in Hystrix config then it should work in zuul
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 07:25
@marcingrzejszczak cool thanks. Let me try this out.
Marcin Grzejszczak
@marcingrzejszczak
Jan 12 2016 07:25
great
hacbq
@hacbq
Jan 12 2016 10:13
my products whole are small files :smile:
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 10:33
@marcingrzejszczak If I understand correctly, I don't need to append RibbonCommand anymore to my service id. Is that correct?
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 11:19
How about spring-cloud/spring-cloud-netflix#254 where it mentioned about we need to use /zuul prefix? So let say my zuul service is available at api.example.com and my upload API at api.example.com/media/upload, to bypass the Spring DispatcherServlet, I need to use api.example.com/zuul/media/upload instead?
Dave Syer
@dsyer
Jan 12 2016 11:20
Yes. That's the feature.
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 11:26
I don't have errors anymore both on Zuul and the microservice, but getting Gateway Timeout 504 error on the consumer. This is my current configuration on Zuul (my microservice name is media):
zuul:
  routes:
    media: /media/**
#  host:
#    connect-timeout-millis: 60000
#    socket-timeout-millis: 60000

media:
  ribbon:
    ReadTimeout: 600000

hystrix:
  command:
    media:
      execution:
        isolation:
          strategy: THREAD
          thread:
            timeoutInMilliseconds: 600000
thanks @dsyer
Dave Syer
@dsyer
Jan 12 2016 11:27
504 is a server side error
Maybe you need to set ConnectTimeout as well?
I'd expect to see a log of the error on the server in any case
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 19:51
@dsyer I try it again, but the behavior is still the same. No errors on both services. The upload finishes in around 100 seconds for a 350 KB file. From Chrome Network tool, I see that the waiting time is exactly 1 minute before it gets the 504. So it seems to me the 10 minutes timeout is not respected here or I configured it wrongly.
Let me try with ConnectTimeout as well. And maybe use default first instead of service id just to see if the timeout works.
Dave Syer
@dsyer
Jan 12 2016 20:14
100s for a 350k file is rubbish bandwidth
Are you sure it's actually downloading all that time?
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 20:34
Actually the 100 seconds is mostly spent by this function where it upload the file into a bucket. (For info, it's a S3 compatible API, not really Amazon S3). My service's upload controller is being called quickly.
getS3Client().putObject(
                            new PutObjectRequest(
                                    connectionSettings.getBucketName(),
                                    file_path,
                                    file.getInputStream(),
                                    new ObjectMetadata()));
What I did is just a simple log before and after this method call and see the timestamps
Fairuz Wan Ismail
@wmfairuz
Jan 12 2016 20:51
@dsyer but I do agree with you that it's rubbish bandwidth