These are chat archives for KnpLabs/Gaufrette

7th
Jun 2016
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 15:57
Hello,
I'm pretty sure this is a recurrent question but... what are the best practices when comes to upload large files to S3 using the awsS3 adapter.
My problem is the memory usage :(
(I'm new at this gitter thing, please excuse me if I'm rude asking directly) :)
Andrew Kovalyov
@akovalyov
Jun 07 2016 17:11
hi Jaime and welcome to gitter :)
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 17:12
:-)
Andrew Kovalyov
@akovalyov
Jun 07 2016 17:14
I assume that you're passing the whole content to write method, right?
for S3 adapter it should work fine if you pass resource (however, I haven't tried it yet and don't have an opportunity to test it right now)
so if you could try it and report if it works, then we can improve some docs and save some time and nerves for Gaufrette users.
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 17:15
Yes, I'm quite sure I am passing the whole content (I use the Gaufrette wrapped in a cakephp plugin)

"pass the resource" I will be happy triying but my question is ¿where?

I Think the first memory kaputt-point is in the _storeFile of the abstractListener. Should I try to rewrite that function? or should I try to create a new listener and rewrite the _storeFile in it?

(excuse my english, it's not my native language :) )
(but I believe that the memory-kaputt-point is a pretty descriptive concept :D )
Andrew Kovalyov
@akovalyov
Jun 07 2016 17:26
no worries. To be honest, I haven't tried neither cakephp nor gaufrette plugin for that.
Are you talking about this ?
I think, that fopen should be there instead of file_get_contents. If it works, please report it here. I think, in this case we can improve both docs and that plugin.
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 17:34
wow, yes. That exact line :)
fopen... ok, i'll try and will be back with the results :)
thanks a lot
Andrew Kovalyov
@akovalyov
Jun 07 2016 17:35
my pleasure.
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 17:36
I just realize that I was talking about the abstractListener as if it comes with the gaufrette library and you already know it. My bad. Sorry.
I have spent several hours digging from my controller to the burzum library down to the gaufrette and hitting the rock bottom in the aws sdk and i'm messing it all in my mind :)
thank again for your patience :)
Andrew Kovalyov
@akovalyov
Jun 07 2016 17:44
no worries, sometimes debugging can be really hard. And I am really interesed to write down all the troubles with memory, because actually it is a common trouble of filesystem abstractions. We are experimenting with ways to fix that, but still it is highly WIP.
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 18:25
Win!!! It seems to work!
Andrew Kovalyov
@akovalyov
Jun 07 2016 18:25
:thumbsup:
Jaime Domínguez
@jdominguezpaz
Jun 07 2016 18:25
I tried 2 things. Pass a resource with fopen as you suggest and pass anGuzzle\Http\EntityBodyInterface object
the later results to slow (200kb upload out of a 20mbps connection)
but the fopen works
but!
to the fopen to work it's required to pass the contentLength to the putObject method so i add this to the gaufrette S3Client
    if (is_resource($options['Body'])) {
        $stats = fstat($options['Body']);
        $options['ContentLength'] = $stats['size'];
    }
in the getOptions function
I'll make more tests but it looks the memory issue is over for me :) Now I can lower that 6gb memory limit cap in the php.ini :)
thanks again!
(to the gaufrette s3 adapter, no s3 client. Sorry) :D