Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 17:37
    SamRemis synchronize #2532
  • 17:32
    SamRemis edited #2532
  • 17:31
    SamRemis opened #2532
  • 16:42
    yenfryherrerafeliz edited #2531
  • 16:42
    yenfryherrerafeliz review_requested #2531
  • 16:41
    yenfryherrerafeliz opened #2531
  • Sep 26 18:15

    aws-sdk-php-automation on master

    Update models for release 3.236.0 release (compare)

  • Sep 26 18:15

    aws-sdk-php-automation on 3.236.0

    (compare)

  • Sep 26 17:54
    SamRemis synchronize #2517
  • Sep 26 13:57
    SamRemis synchronize #2517
  • Sep 26 13:47
    SamRemis synchronize #2517
  • Sep 24 06:13
    getsolaris edited #218
  • Sep 24 06:13
    getsolaris opened #218
  • Sep 23 18:52
    tchernomax commented #2519
  • Sep 23 18:46
    tchernomax synchronize #2519
  • Sep 23 18:19

    aws-sdk-php-automation on master

    Update models for release 3.235.15 release (compare)

  • Sep 23 18:19

    aws-sdk-php-automation on 3.235.15

    (compare)

  • Sep 23 17:07
    yenfryherrerafeliz unlabeled #2524
  • Sep 23 17:07
    yenfryherrerafeliz unlabeled #2524
  • Sep 23 17:07
    yenfryherrerafeliz labeled #2524
Zxurian
@Zxurian
in fact most of them
Jonathan Eskew
@jeskew
Those will be handled as multipart uploads
So it looks like all uploads will start
But multipart uploads may not finish before the end of a script
this is the test I was running:
$s3 = new \Aws\S3\S3Client([
    'region' => 'us-west-2',
    'version' => 'latest',
    'debug' => true,
]);

$uploads = [
    'small' => $s3->uploadAsync(
        'testing-testing-testing',
        'small-file-async-upload',
        fopen(__FILE__, 'r')
    ),
    'large' => $s3->uploadAsync(
        'testing-testing-testing',
        'large-file-async-upload',
        fopen(__DIR__ . '/phparchitect-2015-06.pdf' , 'r'),
        'private',
        [
            'before_upload' => function (\Aws\CommandInterface $command) {
                echo "uploading chunk {$command['PartNumber']}\n";
            }
        ]
    )
];

echo "uploading\n";
\GuzzleHttp\Promise\queue()->run();
The second upload is large enough to trigger a multipart upload
Which is handled by a Guzzle coroutine
And coroutines might not be complete before the call to \GuzzleHttp\Promise\queue()->run() finishes
Zxurian
@Zxurian
hm...
creating a test script now
Jonathan Eskew
@jeskew
does that sound like it’s the same error you’re seeing?
Zxurian
@Zxurian
well mine's not erroring out, it's just never completing
Jonathan Eskew
@jeskew
right
but does that fit the general characteristics of the issue you’re facing?
the script terminates and uploads still haven’t finished
Zxurian
@Zxurian
I'm getting the promise from the uploadAsync(), setting a then() on the promise, then looping waiting for a value to be set from the then(), so my script never terminates
it just keeps waiting for it but never completes, and chekcking separately on S3, nothing ever gets uploaded
Jonathan Eskew
@jeskew
Ok
Guzzle doesn’t have a built in event loop, so coroutines and functions registered with ->then need some prompting
This can either be by calling ->wait, which will unwind a full promise stack
Zxurian
@Zxurian
hm...so if I'm just issuing ->uploadSync() like so
Jonathan Eskew
@jeskew
or by manually ticking the loop
Zxurian
@Zxurian
$promise1 = $S3->uploadAsync('tantor-books', 'Test/test.xml', fopen('/mnt/olympus/Books/0087_CoalHuman/Download/0087_CoalHuman.xml', 'r'));
$promise1->then(function() use ($uploading) {
    $uploading --;
});
it won't automatically start uploading?
Jonathan Eskew
@jeskew
It will
but the upload will kick off additional promises
Zxurian
@Zxurian
that's what I thought. okay, let me run through some additional tests quick
Jonathan Eskew
@jeskew
one of the core developers of React wrote an integration
where the React loop will tick the Guzzle loop
and that can achieve something close to what you’re describing
Zxurian
@Zxurian
thanks, I'll take a look at it
so I'm working with this basic example
```php
<?php

$uploading = 0;

$AWS = new Aws\Sdk([
    'version'    => 'latest',
    'region'    => 'us-east-1',
    'credentials'    => [
        'key'        => '***',
        'secret'    => '***',
    ]
]);

$S3 = $AWS->createS3();

echo "beginning first promise\n";
$uploading ++;
$promise1 = $S3->uploadAsync('tantor-books', 'Test/test.xml', fopen('/mnt/olympus/Books/0087_CoalHuman/Download/0087_CoalHuman.xml', 'r'));
$promise1->then(function() use ($uploading) {
    $uploading --;
});

echo "beginning second promise\n";
$uploading ++;
$promise2 = $S3->uploadAsync('tantor-books', 'Test/test.zip', fopen('/mnt/olympus/Books/0087_CoalHuman/Download/0087_CoalHuman.zip', 'r'));
$promise2->then(function() use ($uploading) {
    $uploading --;
});

echo "beginning loop\n";
while ($uploading != 0) {
    echo 'uploading: '.$uploading."\n";
}
Jonathan Eskew
@jeskew
For that use case, I think it would be easier to collect promises and wait on an array of them
Zxurian
@Zxurian
and just running that, the script will never exit, as it will loop continuously on the last section, as the $uploading value never changes, because the then() functions aren't firing, because the files never upload. I've also let it run and checked externally to see if the files ever showed up on the S3 bucket, and they didn't, which leads me to believe that the uploadAsync() method isn't firing when called
Jonathan Eskew
@jeskew
That while loop won’t give anything a chance to upload
Zxurian
@Zxurian
but you said earlier that issuing an uploadAsync() starts immediately
Jonathan Eskew
@jeskew
But you could collect the promises into an array
it does
However, Guzzle needs a chance to process the response that comes in asynchronously
and in the case of multipart uploads, initiate the upload of different parts of the original file
Zxurian
@Zxurian
so is there any way to start the upload without blocking the parent and have it continue?
Jonathan Eskew
@jeskew
For that you need to manually tick the underlying loop
If you want that to happen in the background, a ticking function needs to be registered in an asynchronous event loop
like React's
Alternatively, you can collect promises into an array
Zxurian
@Zxurian
so if the upload is large enough to trigger a multi-part upload, the uploadAsync() won't really perform it asynchronously as you need to keep ticking it from your main script processing for it to complete to then process the then() attached to the function?
Jonathan Eskew
@jeskew
it will perform some parts of it asynchronously
there’s a call that must be made to initiate a multipart upload