Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Nicolai Kamenzky
    @analog-nico
    AFAIK you21979/node-limit-request-promise is limiting the requests in a general way
    You need to limit based on the responses, right?
    Pulkit Singhal
    @pulkitsinghal
    yes that module seems to be more of a queue to crawl the same domain again and again ... and queues are good ... its better to have queue tied to a timer/scheduler process than a .delay() in nodejs ... but for me to use it ... i would need to submit a PR that reuses the queue to retry failed calls by re-queuing them ... not sure if i'm making sense
    not even sure if there is a real queue underlying you21979/node-limit-request-promise or if i'm just letting the conceptual talk get ahead of me without looking at the code
    anyhow i don't have an action item for you ... just wanted to let you know what went down in the end
    Nicolai Kamenzky
    @analog-nico
    Good to know. BTW, you21979 is a very open minded guy and would probably accept your PR. However, since the approaches are different it would maybe make more sense to write a new lib.
    I will have a similar use case as you have in the near future.
    Keep me in the loop if you write a lib for that etc.
    Pulkit Singhal
    @pulkitsinghal

    Hey @analog-nico , I've recently been running into caught an error: [SyntaxError: Unexpected end of input] more and more for my code which looks like:

    var request =  require('request-promise');
    return request(options)
        .then(successHandler)
    ...
    var successHandler = function(response) {
      if(_.isArray(response)) {
        console.log('response is an array');
      }
      else if(_.isObject(response)) {
        console.log('response is an object');
        return Promise.resolve(response);
      }
      else if(_.isString(response)) {
        console.log('response is a string');
        //console.log(response);
        try{
          var responseObject = JSON.parse(response);
          //console.log(responseObject);
          return Promise.resolve(responseObject);
        }
        catch(error){
          console.log(response);
          console.error('caught an error: ', error);
          throw error;
        }
      }
      else {
        console.log(response);
      }
    };

    Have any of your other users who may be handling large responses, reported anything similar ... is there a better way for me to handle a large response?

    Nicolai Kamenzky
    @analog-nico
    Hi @pulkitsinghal if you are talking about megabytes you maybe should switch to streams (piping). However, then you also need a streaming JSON parser etc. etc.
    Anyway, you might get a good answer in the request/request chat. Request-Promise behaves in your case exactly the same as if you would use Request with a callback.
    From your comments I assume that you looked into the responses. How does one look like if you get a SyntaxError?
    Pulkit Singhal
    @pulkitsinghal
    its like 90% of a json object string
    Nicolai Kamenzky
    @analog-nico
    Weird. Did you try these requests with a independent API test tool?
    Pulkit Singhal
    @pulkitsinghal
    hmm the one that failed was
    28.22 KB (28.94 KB)
    in size
    Nicolai Kamenzky
    @analog-nico
    E.g. with Postman
    Pulkit Singhal
    @pulkitsinghal
    no i have not done that ... before i go that route, can you recommend a good profiler for NodeJs?
    Nicolai Kamenzky
    @analog-nico
    With that you could check whether the error is related to the Request lib or actually in your API.
    Pulkit Singhal
    @pulkitsinghal
    even with my api it works 50% of the time for the exact saem data
    and fails the other 50%
    Nicolai Kamenzky
    @analog-nico
    Sorry, I never used a profiler in node...
    Pulkit Singhal
    @pulkitsinghal
    i'll keep you posted, in case i find a real bug
    thanks!
    Nicolai Kamenzky
    @analog-nico
    Alright. Ping me and I can give you some more tips.
    Travis Bloom
    @travisbloom
    hi @analog-nico, regarding the promise now returning errors on failure, I want to propagate that throw to my express error catcher. If I don't include error, I get a possible unhandled errorStatusCode. Whats the correct way of passing this error back from the promise without having to repeat catch on all the requests?
    Nicolai Kamenzky
    @analog-nico
    hi @travisbloom in most cases you should be able to work with those error objects exactly as you did before when they still were simple objects. Anyway, I don't yet understand your use case. Could you e.g. give me some code snippets?
    Kiệt Thành Võ
    @vothanhkiet
    Hi, I'm using Postman to post a request to API server, it took about <1s, but when I use post request with request-promise, it took to long, can up 20s
    Nicolai Kamenzky
    @analog-nico
    Hi @vothanhkiet maybe its the parameters you use. Can you paste your code snippet?
    Kiệt Thành Võ
    @vothanhkiet
    Srr @analog-nico that's not related to your library, the problem come from DNS resolve, If I use IP address, the response time was fast, but when use domain name, the response time is slow down. Other domain name is ok, but that .aero domain got problem
    Maybe chrome cache DNS so run fast each request. Nodejs dont cache so it take much time each response
    Nicolai Kamenzky
    @analog-nico
    Interesting issue @vothanhkiet Did you decide to just use the IP for now? Or maybe use DNS caching in node? (https://www.npmjs.com/package/native-dns-cache)
    Kiệt Thành Võ
    @vothanhkiet
    I try to avoid use IP for now, if the problem can't solve, then I must use IP instead of domain, 16s is too bad for user experience. Let me try DNS caching in node, thanks for you help :)
    Amanda Harlin
    @amandaharlin
    aw, hey @pulkitsinghal!
    Pulkit Singhal
    @pulkitsinghal
    Sup @amandaharlin - small internet?
    Evgeny Sinitsyn
    @cuhuak
    Hi, dudes
    How do i download a file using rp?
    Nicolai Kamenzky
    @analog-nico
    Hi @cuhuak
    Basically like you would do with Request.
    Is it a small file or something bigger that you prefer to load through a stream?
    Evgeny Sinitsyn
    @cuhuak
    @analog-nico, hi. It is big file (a set of fonts). And yes, i did it with request (not request-promise)
    Nicolai Kamenzky
    @analog-nico
    @cuhuak If you are piping the data you should use the original Request library. Although you can use .pipe() even with Request-Promise you should not pipe much data with it. Rule of thumb is: Use Request-Promise if you would otherwise use Request with a callback.
    BTW, the reason for this is that the whole file contents have to be cached in order to return them in the resolved promise. Though that is technically possible it affects your memory usage. For that reason many prefer streams so that at any time just chunks of the big file is held in memory. Request-Promise always caches the whole file even if you only use .pipe(). It can't foresee if you plan to call .then() or not. ;)
    Evgeny Sinitsyn
    @cuhuak
    @analog-nico , thank you for your answer - very helpful.
    but when i pipe with request, how do I check response.statusCode?
    Nicolai Kamenzky
    @analog-nico
    request
      .get('http://google.com/img.png')
      .on('response', function(response) {
        console.log(response.statusCode) // 200 
        console.log(response.headers['content-type']) // 'image/png' 
      })
      .pipe(request.put('http://mysite.com/img.png'))
    Did you use streams before?
    Evgeny Sinitsyn
    @cuhuak
    @analog-nico , thanks again. yes i did. I definitely should rtfm... thanks!
    Nicolai Kamenzky
    @analog-nico
    You are welcome @cuhuak !
    Samir Prakash
    @samirprakash
    Can anyone help me with setting headers before a GET call is made using request-promise?
    Nicolai Kamenzky
    @analog-nico
    Hi @samirprakash which headers do you need?