Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Philip Ardery
    @arderyp
    yeah, if it is indeed an issue, I think its tangential and doesn't really affect the meat of our issue/pr.
    Michael Butler
    @michaelbutler
    maxHandles 1+ can cause a connection leak because of the >= comparison.
    if you're using the same Client object throughout your application (via injection?) maybe you should consider creating it on the fly and then using it for a few thousand requests, then let it get destroyed
    Philip Ardery
    @arderyp

    You touched on the issue in the last paragraph you just typed...

    In thinking about your solution more, I think there is an inherent issue that makes it incomplete.

    Your scenario seems to be batching sets of requests, and your solution seems to provide a functionality for clearing out the handles after processing each batch. Then you re-initiate the client/factory, process a batch of requests, clear the FD handles, and so on.

    However, for efficiency, some people like myself want to use a single configured client (embedded in some wrapper object) and then issue an unknown dynamic number X requests. Since we don't know how big X is, we'd want to establish a maxHandle limitation that prevents the system from processing too much at once.

    Unfortunately, your solution won't fix this situation, because I may throw 5000 requests at my client object, and I can't call your cleanup until AFTER all of those 5000 requests are processed. My understanding is that the real solution would be for guzzle to know, at all times, how many temporary files are open and which handler points to which temp file. It should then close those handles and temp files when they are no longer needed, and should never allow the number of open temp files (or handlers) to exceed maxHandles. That simply isn't happening now.

    Debugging also suggests that count($this->handles) in CurlHandler does NOT match the number of actually open temp files on the system (I used your gist code for dumping the count alongside the actual FD file count). While the handle count does throttle to fulfill maxHandles, the actual number of open temp files does not.

    I've tried using $easy->sink->close() before/after the curl_close call, but that doesn't work. I also tried setting the raw temp file handler in a local dynamic property on the $easy object around here:

    https://github.com/guzzle/guzzle/blob/master/src/Handler/CurlFactory.php#L400

    then tried calling fclose on it down by the curl_close call, and it didn't work (not a valid resource error).

    Michael Butler
    @michaelbutler
    from what I can tell, for each request there should only ever be at most 2 FDs: one for the curl handler (returned from curl_init) and one for the response body (a resource stream, could be like an fopen handle to a temp file). so in your specific case the code would have to be investigated
    Philip Ardery
    @arderyp
    this issue is really frying my brain haha
    Michael Butler
    @michaelbutler
    but to be honest I don't know what you mean by dynamic number X requests. It wouldn't be hard to create a class that accepts an arbitrary number of URLs as input, and have a while loop that batches them up , say 100 at a time, destroying the memory (and handles) after each 100, until the original list becomes empty
    Philip Ardery
    @arderyp

    I agree with your last comment, but i don't understand "your specific case the code would have to be investigated".

    Do you not agree that its possible to give a single guzzle client too many request, at which point it will buckle and show the failed to open stream: Too many open files error?

    You solution is nice for certain use cases, but it doesn't prevent the client from reaching that error. And the client reaches that error (IMO) because guzzle isn't properly managing temp file garbage collection.

    that count($this->handles) doesn't match the FD count that your gist produces demonstrates that
    Michael Butler
    @michaelbutler
    yes, in my opinion if you make too many concurrent requests (i.e. if the code sample on http://docs.guzzlephp.org/en/stable/quickstart.html#concurrent-requests had 2000 requests in the $promises array) you can hit Too many open files error
    Philip Ardery
    @arderyp
    right
    okay
    Michael Butler
    @michaelbutler
    nothing can really be done there on the Guzzle side
    the promises need to be at least periodically settled otherwise it's going to try to keep ALL the requests open in memory and FDs at the same time
    my issue is that originally my code WAS processing stuff in batches, and yet we still were hitting the too many open files issue
    1000 is quite a lot of HTTP requests to make in parallel. web browsers are typically limited to around... 20 I think
    furthermore you could also stress the system on the other end depending on their limits
    Philip Ardery
    @arderyp

    Yes, your specific issue is definitely a problem, and your solution is good for that. By the way, should your cleanup logic not just go into the CurlHandler destruct, so you don't have to call it each time?

    All that being said, my ultimate goal would be to improve the default handler logic to be able to keep precise track of open files, sloce them when they are not necessary, and prevent the number of open files from ever exceeding maxHandles. Then, if the user ever reached the aforementioned server error, all they'd have to do is provide an appropriate maxHandle number that their server can handle.

    but, maybe you are right, and I (and others) simply shouldn't send an obscene number of requests to a client. But even in that case, I think implementing the above would be beneficial because there is clearly lots of confusion on this issue. "maxHandles" seems to suggest that no more than maxHandles connections will ever be open at a time. As it is currently implemented, it means that no more than maxHandle CURL handlers will be open at a time, but temp file handles can (and likely do) exceed maxHandle, often by a lot. Maybe there should be a maxFileHandler argument thats totally separate from maxHandle that ensures that I am talking about?

    Michael Butler
    @michaelbutler
    i think the problem with keeping track of open files and automatically closing them is that it would break a lot of stuff. if you want to make 5000 requests at once, that will correspond to 5000 responses, each with an open FD. If guzzle auto closed 4000 of them before your code even had a chance to read the response, there would be a lot of unhappy engineers!
    For putting the cleanup in the __destruct, I think I tried that before but the problem was chicken-and-egg; PHP was not invoking the __destruct because of the curl connection that still wasn't closed. you have to close it first somehow
    Philip Ardery
    @arderyp
    hmm, maybe you are right
    about my idea for new implementation
    Michael Butler
    @michaelbutler
    and yeah maxHandles is misleading. that's just for curl connections, not the response body handles which is handled somewhere else
    Philip Ardery
    @arderyp
    Do you think there is some way to get your approach to work in __desctruct or do you think its impossible given the logic? Also, I should take the time to THANK YOU immensely for engaging with me on this. Sadly, the Guzzle devs dont seem interested.
    Michael Butler
    @michaelbutler
    if I have a chance I can try using __destruct again for the cleanup instead of manually calling it. should be provable in the gist
    and yeah, no problem.
    Philip Ardery
    @arderyp
    Many thanks Michael. I will continue to watch this. I think a destruct solution would be ideal because then it would happen behind the scenes without the implementing user having to worry about it. But if its impossible, your approach is nice.
    Michael Butler
    @michaelbutler
    👍🏼
    Raj
    @cyrus19901
    Hi all, new to Guzzler or in general PHP. I am working on creating GET/POST request using send async. I am confused on how to add headers and body to the request. `$headers = array('Api-Timestamp'=> $timeStamp, 'Api-Key'=> $this->apiKey,'Accept'=> 'application/json');
    $body = $content;
    $options =array('headers'=>$headers,'body'=>$body);
    // $request = new \GuzzleHttp\Psr7\Request('GET', $uri, $options);
    $client = new \GuzzleHttp\Client();
        # Send an asynchronous request.
        print_r($options);
        $request = new \GuzzleHttp\Psr7\Request($method, $uri,$options);
        $promise = $client->sendAsync($request)->then(function ($response) {
            echo 'I completed! ' . $response->getBody();
        });` I am not getting the right response with the headers but once i remove  the headers i  get the response back. Is there something i am missing out. You help will be much appreciated .
    adira gasada
    @Gar0nGs_twitter
    how to install in windows 10
    duncz
    @duncz
    morning folks. I'm having difficulty getting my head around Pool requests, and how I can catch their exceptions. I put a thread on Stack Overflow, but it hasn't drawn much attention and I don't have enough points to raise a bounty. could someone please have a look and let me know their thoughts? https://stackoverflow.com/questions/56183592/handling-exceptions-on-a-pool-of-requests
    Hayden Young
    @haydenyoung

    Hi. I am attempting to use Guzzle to consume this service: https://github.com/orbitdb/orbit-db-http-api.

    However, when passing stream = true to the client get method, I receive the error:

    <GET https://localhost:3000/db/my-feed/events/load> [CONNECT] 
    <GET https://localhost:3000/db/my-feed/events/load> [AUTH_RESULT] severity: "2" message: "HTTP/1.0 403 Forbidden
    " message_code: "403" 
    <GET https://localhost:3000/db/my-feed/events/load> [MIME_TYPE_IS] message: "text/plain" 
    <GET https://localhost:3000/db/my-feed/events/load> [PROGRESS] 
    PHP Fatal error:  Uncaught GuzzleHttp\Exception\ClientException: Client error: `GET https://localhost:3000/db/my-feed/events/load` resulted in a `403 Forbidden` response in /home/html/orbitdb/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113

    However, if I remove stream true, it looks fine (debug=true enabled on request):

    *   Trying 127.0.0.1...
    * TCP_NODELAY set
    * Connected to localhost (127.0.0.1) port 3000 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * successfully set certificate verify locations:
    *   CAfile: /etc/ssl/certs/ca-certificates.crt
      CApath: /etc/ssl/certs
    * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
    * ALPN, server accepted to use h2
    * Server certificate:
    *  subject: C=AU; ST=A; L=B; O=Organization; OU=OrganizationUnit; CN=localhost; emailAddress=demo@example.com
    *  start date: May 25 14:56:35 2019 GMT
    *  expire date: Oct  6 14:56:35 2020 GMT
    *  common name: localhost (matched)
    *  issuer: C=AU; ST=A; L=B; O=Internet Widgits Pty Ltd; CN=Local Certificate
    *  SSL certificate verify ok.
    * Using HTTP2, server supports multi-use
    * Connection state changed (HTTP/2 confirmed)
    * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    * Using Stream ID: 1 (easy handle 0x558b8007c8a0)
    > GET /db/zdpuAoZQznotzijaUCrgUEQkPVCH64RCmGHPXUnRPX5EQn3L1%2Fmy-feed2/events/replicate.progress HTTP/2
    Host: localhost:3000
    Accept: text/event-stream
    Cache-Control: no-cache
    User-Agent: GuzzleHttp/6.3.3 curl/7.58.0 PHP/7.2.17-0ubuntu0.18.04.1
    
    * Connection state changed (MAX_CONCURRENT_STREAMS updated)!
    < HTTP/2 200 
    < content-type: text/event-stream; charset=utf-8
    < content-encoding: identity
    < cache-control: no-cache
    < vary: accept-encoding
    < date: Sat, 25 May 2019 15:10:38 GMT

    But without stream true I can't access the network stream. I'm using a self signed cert but have generated a local CA cert which I have added to the local CA store. curl and guzzle seem to be able to resolve the cert without problem (I'm not disabling it using flags like curl -k).

    phillmac
    @phillmac
    ^ seems like 'stream'=>true forces http1.1. which the server does not accept. Where would I look for that piece of code I wonder?
    Hayden Young
    @haydenyoung
    @phillmac seems like there is an option stream_context which can be passed to the request. Wondering if this can be used to force http/2 by the client? https://guzzle.readthedocs.io/en/latest/faq.html?highlight=stream_context#how-can-i-add-custom-stream-context-options
    phillmac
    @phillmac
    It looks too me like the http context has a protocol_version that accepts either 1 or 1.1
    phillmac
    @phillmac
    It might accept 2?But I cannot find any mention in the docs
    Kieron Wiltshire
    @KieronWiltshire
    Guys, guzzle doesn't actually work? it doesn't send form_params
    Sergey
    @NBZ4live
    Hi. I have problems with Guzzle 6.3.3. I’m getting segfaults when I’m trying to make a request.
    (new Client())->post('https://example.com‘);
    I’m getting: [1] 19546 segmentation fault php artisan test.
    Running PHP 7.2.19 on MacOS Mojave.
    I’ve already googled, but did not found anything related to my case.
    martinmike2
    @martinmike2
    hi there, is it possible to use guzzle without a scheme? my use case is this: I am running several php microservices on kubernetes. the k8s internal dns reveals FQDN's such as 'service.namespace.cluster.local' . When I try to point guzzle to that fqdn without a schema (non provided by dns) it keeps the first letter of the domain then strips out the rest.
    for example REQ: 'users-service.users.cluster.local/auth' results in 'uauth'
    phillmac
    @phillmac
    @martinmike2 Are you trying to make just plain http requests? Or naked tcp connections? what are you trying to achieve by omitting the scheme?
    Márk Sági-Kazár
    @sagikazarmark
    Sounds weird to me. Which version of Guzzle do you use? Do you manually provide the URL? Can you try using plain cURL to see the answer? (BTW I think the correct URL is service.namespace.svc.cluster.local)
    Max
    @specialtactics
    Hi there
    Zerox Millienium
    @zeroxmxi
    is it possible to construct raw http requests using Guzzle so they can used to build OData $batch requests?
    ZhangDi
    @zd-double
    @yelena870513 I have the same problem cURL error 0: The cURL request was retried 3 times and did not succeed. The most likely reason for the failure is that cURL was unable to rewind the body of the request and subsequent retries resulted in the same error. Turn on the debug option to see what went wrong. See https://bugs.php.net/bug.php?id=47204 for more information. (see http://curl.haxx.se/libcurl/c/libcurl-errors.html), can somebody help me on this issue? env: (GuzzleHttp/6.3.3 PHP/5.6.23, redhat)
    Abishek R Srikaanth
    @abishekrsrikaanth
    Could anybody advice on how I can convert this to using guzzlephp https://gist.github.com/abishekrsrikaanth/49ddecc1df2bedf576e0f53e66b54bf9.
    Delf Tonder
    @leberknecht
    ahoi! does anyone know/understand/can-explain why guzzle is turning HTTP errors (400-range) into exceptions?
    Delf Tonder
    @leberknecht
    ah, found that i can set 'http_errors' => false to bypass this behavior :)