These are chat archives for nextflow-io/nextflow

14th
Mar 2017
amacbride
@amacbride
Mar 14 2017 18:00
Boy, S3 is having another really bad day. I'm getting a bunch of failures (through the foreign file interface), because of S3 timeouts:
 Timeout waiting for connection from pool
I'm getting the same sort of thing from S3 in general, so I know it's not NF, but I'm still trying to figure out ways around it. Maybe factor out the downloads into an explicit process that I can set a higher retry value for? Any suggestions?
Paolo Di Tommaso
@pditommaso
Mar 14 2017 18:02
umm bad
what if you download that files ?
Mike Smoot
@mes5k
Mar 14 2017 18:08
@amacbride I've had luck doing what you suggest and running aws s3 sync in a process. I've done this in conjunction with the storeDir directive to cache local copies of databases, but I think using a process just for retry capability is not a bad idea.
amacbride
@amacbride
Mar 14 2017 18:26
As background, this is a process that downloads 64 FASTQ files from S3 to the processing instance (EC2 or local). I had originally used the aws sync option, but in general, the foreign-file interface had been working well, and was simpler.
I was wondering if there might be retry options built into the FFI (or if they might be added).
Of course, most of the time it isn't an issue, but I'm looking at edge cases and reliability.
Paolo Di Tommaso
@pditommaso
Mar 14 2017 18:51
well, I think retry it should be managed at the level of the S3 client and actually I think it's doing that
Paolo Di Tommaso
@pditommaso
Mar 14 2017 19:02
look for DEFAULT_MAX_ERROR_RETRY at this page
you can try to increase that value, see
amacbride
@amacbride
Mar 14 2017 19:08
aha! perfect
Paolo Di Tommaso
@pditommaso
Mar 14 2017 19:09
though it looks strange that 3 is not enough
amacbride
@amacbride
Mar 14 2017 19:09
@pditommaso As an aside, looking at this, is is possible to set uploadStorageClass to STANDARD_IA ?
(FYI, REDUCED_REDUNDANCY is about to be deprecated -- Amazon will be phasing it out soon)
Paolo Di Tommaso
@pditommaso
Mar 14 2017 19:13
yes, it should be possible
uploadStorageClass is specified as a string constant
amacbride
@amacbride
Mar 14 2017 19:14
excellent!
Paolo Di Tommaso
@pditommaso
Mar 14 2017 19:15
let me know if increasing the max retries result in any benefit
amacbride
@amacbride
Mar 14 2017 19:15
As to your comment, 3 is almost always enough (I've downloaded hundreds or thousands of files this way over the last few months), but when S3 is having a bad day, it's nice to be able to bump it up.
Will do!
Paolo Di Tommaso
@pditommaso
Mar 14 2017 19:39
makes sense