These are chat archives for request/request

18th
Jan 2015
Harris Robin Kalash
@harrisrobin
Jan 18 2015 14:42
Hi , having a little problem with request + async
I would expect that offset keeps incrementing and it keeps running the createOrUpdate function until it reaches 501, but it only reaches the initial 50 received from the call and then stops.
im sure im making a stupid mistake but been stuck on this for a while
basically, items (the array that gets returned) has 50 items, and it stops after saving those 50.. but i want it to keep incrementing the offset in the url parameter until 501
hence async.whilst
simo
@simov
Jan 18 2015 14:48
you need to put the request call into a function that you call each time you increment the offset
Harris Robin Kalash
@harrisrobin
Jan 18 2015 14:48
ah.. that makes total sense lol
thanks
missing code : var offset = 0;
var totalCount = 501;
still can't get it to run the request until offset is 501
simo
@simov
Jan 18 2015 15:07
function startScraping (done) {
  request('...'+offset, function () {
    createOrUpdateItems(body.items, done)
  })
}

function createOrUpdateItems (items, done) {
  async.each(items, function (item, done) {
    // save
    done()
  }, done)
}

async.whilst(keepScraping, startScraping, function () {
  // end
})
something like that (pseudo code)
Harris Robin Kalash
@harrisrobin
Jan 18 2015 15:38
@simov , finally figured it out thanks
it works
last detail though, for some reason when keepScraping becomes false it keeps running despite the fact that there are no more items to scrape
but other than that, finally got it working :D
thanks again