Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Feb 02 19:45

    aws-sdk-php-automation on master

    Update models for release 3.258.2 release (compare)

  • Feb 02 19:45

    aws-sdk-php-automation on 3.258.2

    (compare)

  • Feb 01 21:38
    stobrien89 opened #2632
  • Feb 01 20:53
    github-actions[bot] commented #2237
  • Feb 01 20:53
    stobrien89 closed #2237
  • Feb 01 20:53
    stobrien89 commented #2237
  • Feb 01 20:52

    stobrien89 on master

    fix: GetLogEvents auto-paginati… (compare)

  • Feb 01 20:52
    stobrien89 closed #2449
  • Feb 01 19:25

    aws-sdk-php-automation on master

    Update models for release 3.258.1 release (compare)

  • Feb 01 19:25

    aws-sdk-php-automation on 3.258.1

    (compare)

  • Feb 01 17:06
    stobrien89 synchronize #2449
  • Feb 01 13:57

    SamRemis on master

    bugfix: signature provider when… (compare)

  • Feb 01 13:57
    SamRemis closed #2631
  • Feb 01 00:04
    github-actions[bot] labeled #2565
  • Feb 01 00:04
    github-actions[bot] commented #2565
  • Jan 31 23:23
    stobrien89 synchronize #2631
  • Jan 31 23:16
    stobrien89 synchronize #2449
  • Jan 31 23:13
    stobrien89 opened #2631
  • Jan 31 23:13
    stobrien89 review_requested #2631
  • Jan 31 21:08
    SamRemis commented #2627
Zxurian
@Zxurian
@wcyrek-comrise excellent, do you have a link referencing that feature, or API?
Witold Cyrek
@wcyrek-comrise
I mean I am sure there is a way, but you have to probably put it in your script using soem sort of concurrency library.
I do not think there is a built in way.
Let me jsut make sure by rummiging through the docs, sicne there might be async way of doign that
Witold Cyrek
@wcyrek-comrise
@Zxurian https://aws.amazon.com/articles/1904 suggests using multi-threading to increase throughput
which would make thing that it is not built in, but like I said you can create multipel threads to process the requests concurrently
Zxurian
@Zxurian
okay, so there's no built-in method with the aws-sdk-php library to fork/queue uploads?
Witold Cyrek
@wcyrek-comrise
That appears to be the case. But you can roll one yourself. Or do a little more digging and find one. I foudn one in python, someone no doubt did something somewhere in PHP.
Zxurian
@Zxurian
thanks. and just to double check, is there another library of aws-sdk besides php that would support pre-generating a list of transfers, and then just pass it to an sdk api and it would take care of uploading everything?
Witold Cyrek
@wcyrek-comrise
I don't think so, the guy who I found used the python SDK, he still had to write his won multithreaded code: https://gist.github.com/jeffaudi/f62a57f11a594e41d11d
Only concurrency that I seen is built-in is for uploading big files where the source is a single file
Zxurian
@Zxurian
thanks, I'll look around and see what's available. I'd rather create a single thread that can parse a list of downloads and let it run, rather than having the active code have to loop over it
Jonathan Eskew
@jeskew
@Zxurian have you taken a look at Aws\S3\MultipartUploader?
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html
It has configurable concurrency (managed by curl multi, not threads) and can return a promise if you want the upload to resolve in the background
Jonathan Eskew
@jeskew
There's a similarly asynchronous way to upload directories as well
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-transfer.html
Zxurian
@Zxurian
@jeskew awesome, that's exactly what I was looking for. Thanks for the info
YohaanYoon
@YohaanYoon
Using AWS SDK PHP, I am trying to send email asynchronously through AWS SES in my local machine (mac, apache)
While $promise->wait() works well, $promise->then doesn't work
Do you have any ideas to fix it? Such as requirement issues, environment issues, and so on.
Robert Casanova
@robertcasanova
Hi guys, sorry to disturb you. I'm having a problem with uploading image to S3. I'm using Sonata Media Bundle inside a PHP project with a multi docker container configuration. I'm having this error
cURL resource: Resource id 54; cURL error: Could not resolve host: s3-eu-west-1.amazonaws.com (cURL error code 6). See http://curl.haxx.se/libcurl/c/libcurl-errors.html for an explanation of error codes.
does somebody have the same problem or some suggestion for this?
Jonathan Eskew
@jeskew
@YohaanYoon could you open an issue on GitHub with a code sample?
@robertcasanova that looks like an outbound networking issue. Are you sure the docker container hosting the app can contact the internet?
Robert Casanova
@robertcasanova
@jeskew thx for your answer. I tryed to reboot and then it works: it happens randomly. Probably, as you said, is something with docker. Thx.
mikemherron
@mikemherron
Hi - I've got a quick question about the automatic handling of ProvisionedThroughputExceededExceptions when writing items to DynamoDB - my understanding is that the SDK handles these automatically, but on my test server, I can see these are being thrown from the SDK. I'm intentionally trying to test this behaviour so I've provisioned a table with 1 Unit of Read Capacity and 1 Unity of Write Capacity, and have a script that will definitely exceed this. I suspect that I'm reaching some internal retry limit, but I've not been able to find documentation on this - can anyone point me to whereabouts in the code the retry logic is handled so I can confirm?
Jonathan Eskew
@jeskew
@mikemherron retries are handled by https://github.com/aws/aws-sdk-php/blob/master/src/RetryMiddleware.php
Dynamo calls are retired 11 times by default
mikemherron
@mikemherron
That's great, thank you for the pointer.
mikemherron
@mikemherron
@jeskew Another quick question if you have time? So I added the below line to RetryMiddleware.php:

$command['@http']['delay'] = $delay($retries++);

  • \Log::info("Retrying with delay:" . $command['@http']['delay'] );

return $handler($command, $request)->then($g, $g);

and observed the below output:
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0.05
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0.1
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0.2
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0.4
[2015-11-05 16:28:48] test.INFO: Retrying with delay:0.8
[2015-11-05 16:28:48] test.INFO: Retrying with delay:1.6
[2015-11-05 16:28:48] test.INFO: Retrying with delay:3.2
[2015-11-05 16:28:48] test.INFO: Retrying with delay:6.4
[2015-11-05 16:28:48] test.INFO: Retrying with delay:12.8
[2015-11-05 16:28:48] test.INFO: Retrying with delay:25.6
I assume the delays are in ms ?
Jonathan Eskew
@jeskew
I believe they're in seconds, though I'll have to check to be sure
The requests have an exponentially increasing delay between them
So that throttled requests will try again after the throttle has been lifted
mikemherron
@mikemherron
Ok thanks - from the timestamps recorded in my logs, it looks like the retries are all being logged at the same second.
In fact, thinking about it now, a retry of "0.05" ms probably doesn't make sense :)
Also, my test script definitely didn't show a delay at all - it just fails when my server returns the 500 due to the uncaught ProvisionedThroughputExceededException.
Jonathan Eskew
@jeskew
Oh so the delay is being calculated but not applied?
mikemherron
@mikemherron
That looks possible - I'm trying to look through the code now, but it's my first time through it so taking me a while!
Jonathan Eskew
@jeskew
Could you open an issue with that log on github?
Might help to get more eyes on it
mikemherron
@mikemherron
Sure
I can see in Waiter.php::getArgsForAttempt() the delay is multipled by 1000 with the comment " // Set the delay. (Note: handlers except delay in milliseconds.)"
I may be in the wrong place entirely, but I wonder if that scaling is being by-passed due to the way that the RetryMiddleware applies the delay
Of course, I might just be in the wrong place altogether.