These are chat archives for aws/aws-sdk-php

12th
Oct 2015
Zxurian
@Zxurian
Oct 12 2015 13:56
Is there a way to stage multiple transfers to s3, or fork/queue the upload process so the script isn't waiting for a transfer to complete before moving onto the next section of the code?
Witold Cyrek
@wcyrek-comrise
Oct 12 2015 13:58
Yes!
Zxurian
@Zxurian
Oct 12 2015 13:58
ex. I have multiple separate directories that need to be uploaded that I can loop on, however I'd rather not wait for the transfer to complete before moving onto the next iteration.
@wcyrek-comrise excellent, do you have a link referencing that feature, or API?
Witold Cyrek
@wcyrek-comrise
Oct 12 2015 13:59
I mean I am sure there is a way, but you have to probably put it in your script using soem sort of concurrency library.
I do not think there is a built in way.
Let me jsut make sure by rummiging through the docs, sicne there might be async way of doign that
Witold Cyrek
@wcyrek-comrise
Oct 12 2015 14:06
@Zxurian https://aws.amazon.com/articles/1904 suggests using multi-threading to increase throughput
which would make thing that it is not built in, but like I said you can create multipel threads to process the requests concurrently
Zxurian
@Zxurian
Oct 12 2015 14:10
okay, so there's no built-in method with the aws-sdk-php library to fork/queue uploads?
Witold Cyrek
@wcyrek-comrise
Oct 12 2015 14:12
That appears to be the case. But you can roll one yourself. Or do a little more digging and find one. I foudn one in python, someone no doubt did something somewhere in PHP.
Zxurian
@Zxurian
Oct 12 2015 14:14
thanks. and just to double check, is there another library of aws-sdk besides php that would support pre-generating a list of transfers, and then just pass it to an sdk api and it would take care of uploading everything?
Witold Cyrek
@wcyrek-comrise
Oct 12 2015 14:18
I don't think so, the guy who I found used the python SDK, he still had to write his won multithreaded code: https://gist.github.com/jeffaudi/f62a57f11a594e41d11d
Only concurrency that I seen is built-in is for uploading big files where the source is a single file
Zxurian
@Zxurian
Oct 12 2015 14:27
thanks, I'll look around and see what's available. I'd rather create a single thread that can parse a list of downloads and let it run, rather than having the active code have to loop over it
Jonathan Eskew
@jeskew
Oct 12 2015 16:09
@Zxurian have you taken a look at Aws\S3\MultipartUploader?
It has configurable concurrency (managed by curl multi, not threads) and can return a promise if you want the upload to resolve in the background
Jonathan Eskew
@jeskew
Oct 12 2015 16:15
There's a similarly asynchronous way to upload directories as well
Zxurian
@Zxurian
Oct 12 2015 18:04
@jeskew awesome, that's exactly what I was looking for. Thanks for the info