These are chat archives for rosshinkley/nightmare

27th
Jun 2016
Mingsterism
@mingsterism
Jun 27 2016 07:12
@rosshinkley hey ross. I'm having trouble ending nightmare in a catch statement. I tried many different ways, but it does not seem to end.
any thoughts?
function runScript(url) {
    nightmare
        .goto(url)
        .evaluate(function)
        .then(function)
        .catch(function(err) {
            console.error(err) // <<< This gets called.
            return nightmare.end() // <<< this does not work. 
        })
        .then()
        .catch(function(err){})  // <<< this does not get called.
}

runScript('http://www.faaaaacebook.com') // wrong url
Rob Brackett
@Mr0grog
Jun 27 2016 14:51

@mingsterism Two things:

  1. That last catch() shouldn’t get called, so, technically, everything sounds a-ok there. When you handle a promise error via catch() or the second argument to then(), that resolves things in the promise chain so they are no longer in an error state (if you throw an error or return a promise that rejects from catch(), though, that will put the promise chain back in to an error state).

It might help to think of it like synchronous code. This:

.then(function(result) {
  // do something A
})
.catch(function(error) {
  // handle error 1
})
.then(function(result) {
  // do something B
})
.catch(function(error) {
  // handle error 2
})
Is kind of like:
try {
  try {
    // do something A
  }
  catch(error) {
    // handle error 1
  }
  // do something B
}
catch(error) {
  // handle error 2
}
Two catch calls in a row will never handle the same error unless the first one re-rejects or throws the error it received.
Rob Brackett
@Mr0grog
Jun 27 2016 14:56
\2. Are you sure the nightmare.end() call is not working, or was that just confusion from the above issue? Everything you're doing looks correct there.
If you have nightmare set to show the window, does it close?
Mingsterism
@mingsterism
Jun 27 2016 14:58
@Mr0grog hi Rob. thanks for the clear explaination. i didnt know that the second .then() function was nested as per your try catch code.
does it mean that if i have 5 .then() means there are 5 nested levels?
Rob Brackett
@Mr0grog
Jun 27 2016 14:59
Yeah, it's much better to think of things that are in sequence for promises as acting like nesting for synchronous code
Mingsterism
@mingsterism
Jun 27 2016 14:59
should my `return nightmare.end() be in the catch statement?
Rob Brackett
@Mr0grog
Jun 27 2016 14:59
yeah, that's totally fine
BUT
if you want to make sure you actually end the nightmare instance, you need it in your then also
because only then OR catch will run
(assuming you don't also have an error in your then, of course)
Mingsterism
@mingsterism
Jun 27 2016 15:02
you mean
.evaluate(function) // << error here
.then(function()) // << does not get called
.catch( return nightmare.end()) 
.then(return nightmare.end())
Rob Brackett
@Mr0grog
Jun 27 2016 15:02
hmmm, well, that will call end() more than once
I mean:
.evaluate(something) // maybe error, maybe not
.then(function(result) {
  // do what you need with result, then:
  return nightmare.end();
})
.catch(function() {
  return nightmare.end()
})
Mingsterism
@mingsterism
Jun 27 2016 15:04
oh. but i only wanted to end nightmare if there was error though. if there is no error, i want it to continue.
Rob Brackett
@Mr0grog
Jun 27 2016 15:04
oh, then don't put it in the then :)
Mingsterism
@mingsterism
Jun 27 2016 15:04
so just catch?
Rob Brackett
@Mr0grog
Jun 27 2016 15:04
yeah
I thought you were trying to clean up when you were all done
Mingsterism
@mingsterism
Jun 27 2016 15:05
yeah, but now im testing first with error situation.
then later will place it in end.
Rob Brackett
@Mr0grog
Jun 27 2016 15:07
:thumbsup:
Mingsterism
@mingsterism
Jun 27 2016 15:08
@Mr0grog wondering could you give me feedback on my code.
what do you think of it
its a very rudementary code. how can i structure it for scale, readability, tdd etc... any thoughts appreciated.
also , the problem i mentioned earlier was in line 58. the ERRRRRRRRRRRRRRR is called. but nightmare instance does not close.
Rob Brackett
@Mr0grog
Jun 27 2016 15:14
oh, that's probably because nightmare is not a local variable there
instead of actually calling nightmare.end() it is probably throwing an error
Mingsterism
@mingsterism
Jun 27 2016 15:15
line 57 gets called
Rob Brackett
@Mr0grog
Jun 27 2016 15:15
you need self.nightmare.end()
Mingsterism
@mingsterism
Jun 27 2016 15:15
ah i see.
my electron window closes but on my console, it does not go to commandline. i must ctrl-c. does it mean nightmare is still running?
rosshinkley @rosshinkley pokes head in
Ross Hinkley
@rosshinkley
Jun 27 2016 15:17
@mingsterism you're not closing the mongoose connection
probably why your application isn't ending
sorry to butt in :)
Mingsterism
@mingsterism
Jun 27 2016 15:18
no worries Ross. always appreciate your help. :)
any thoughts in regards to how the code can be improved?
so can go in the right direction to improve
Ross Hinkley
@rosshinkley
Jun 27 2016 15:20
hm, i still think getTitlesAndNextUrl shouldn't be exposed on an instance
Mingsterism
@mingsterism
Jun 27 2016 15:21
yes. sry. will have to get that fixed. but other improvements.
Ross Hinkley
@rosshinkley
Jun 27 2016 15:21
ah, that's fine
beyond that... i don't want to muddy up your process too much
Mingsterism
@mingsterism
Jun 27 2016 15:24
ah. thanks. but was thinking of future scripts though.
Ross Hinkley
@rosshinkley
Jun 27 2016 15:24
you might want to take a look at how nightmare-swiftly works, and how it uses .use()to create reusable action chains
Mingsterism
@mingsterism
Jun 27 2016 15:24
alright.
Ross Hinkley
@rosshinkley
Jun 27 2016 15:24
(it's a little out of date)
(but it also seems to be in the same spirit of what you're doing with respect to scraping/automation)
Mingsterism
@mingsterism
Jun 27 2016 15:25
haha. yeah. i see last commit was 2 years ago :smile:
cool.
Rob Brackett
@Mr0grog
Jun 27 2016 15:26
.then(function(v) {
    console.log(v.nextUrl, '===========++++++')
    Promise.all(v.titles.map(function(title) {
        new self.model({job: title}).save((err, result) => {
            if (err) console.error(err);
            console.log(result);
        })
    }))
    return {
        nextUrl: v.nextUrl
    }
})
Could just be
.then(function(v) {
    console.log(v.nextUrl, '===========++++++')
    return Promise.all(v.titles.map(function(title) {
        new self.model({job: title}).save()
    })).then(function() {
      return {nextUrl: v.nextUrl}
    });
})
Since you are already catching and logging errors in the next step
you could also just return v.nextUrl instead of an object with only that in it return {nextUrl: v.nextUrl}
rosshinkley @rosshinkley nods
Rob Brackett
@Mr0grog
Jun 27 2016 15:29
(Or, if you don't mean to wait for the saves to complete, there's no reason to use Promise.all there at all)
Mingsterism
@mingsterism
Jun 27 2016 15:30
@Mr0grog wondering is there a difference apart from readability in regards to nesting a .then() inside the outer .then(function(v)
Rob Brackett
@Mr0grog
Jun 27 2016 15:31
yes and no…
Mingsterism
@mingsterism
Jun 27 2016 15:34
no in regards to?
Rob Brackett
@Mr0grog
Jun 27 2016 15:34
if going for readability, (and reusability!) I think the bigger thing is to package that up in a function that you can give a descriptive name to:
function saveTitles(data) {
  return Promise.all(data.titles.map(function(title) {
      new self.model({job: title}).save()
  }))
}

// later...
.then(saveTitles)
or later:
.then(function(data) {
  return saveTitles(data).then(Promise.resolve(data.nextUrl));
})
but one way or the other, you kind of need to do that if you want to wait for the saves to complete
Mingsterism
@mingsterism
Jun 27 2016 15:37
interesting. yeah it makes sense to make functions from that.
Rob Brackett
@Mr0grog
Jun 27 2016 15:38
the real idea with promises is actually that you shouldn't be calling then() much
you should just be passing the promises around
but when promises got added to JS, they didn't come with the usual types of helpers that the languages they came from had for that
so ¯_(ツ)_/¯ you kind of have to do more yourself
Mingsterism
@mingsterism
Jun 27 2016 15:39
haha. i guess so. thanks for that advise. it will really help me structure my code better.
wondering what do you think of reactive programming
is it a complement or alternative to promises / generators?
rsjs
rxjs
Rob Brackett
@Mr0grog
Jun 27 2016 15:40
Other super minor nits:
  1. Get those require() calls out of the constructor
  2. Because Nightmare has support for newer APIs, Array.prototype.slice.call(document.querySelectorAll(...)) can just be Array.from(document.querySelectorAll())
oh, reactive is very nice
streams are great
I consider them a complement to promises
Streams are like the array version of promises
Much less convenient if you are only trying to get one value (promises are better for that), but very nice otherwise
would be like having:
function sum(a, b) {
  return [a + b];
}
But for lists of things or endless series of things, like events, which Rx* is mostly focused on, I think it's pretty awesome
Very similar to promises, in my opinion
Mingsterism
@mingsterism
Jun 27 2016 15:43
is streams part of reactive? are they similar to observables?
Rob Brackett
@Mr0grog
Jun 27 2016 15:43
observables are streams
just under a different name with more useful tools
but the basic concept is exactly the same
Mingsterism
@mingsterism
Jun 27 2016 15:44
in the case of web scraping tasks, is reactive useful?
Rob Brackett
@Mr0grog
Jun 27 2016 15:44
There is also http://highlandjs.org, which is a similar idea, but more narrowly-focused

in the case of web scraping tasks, is reactive useful?

Maybe. I don't think it's utility is particular to scraping

Dmitri
@katabatic
Jun 27 2016 15:46
hello
Rob Brackett
@Mr0grog
Jun 27 2016 15:46
if your use case is something like "scraping a long list of URLs" or "URLs will regularly come in and we need to scrape them as they arrive" it definitely would be
hullo!
Ross Hinkley
@rosshinkley
Jun 27 2016 15:46
greetings :)
Dmitri
@katabatic
Jun 27 2016 15:46
:)
Ross Hinkley
@rosshinkley
Jun 27 2016 15:47
@Mr0grog or if you're doing a polling mechanism looking for changes
Rob Brackett
@Mr0grog
Jun 27 2016 15:47
^ that's a good one!
lots of great use cases for Rx/Rx-esque tools
Ross Hinkley
@rosshinkley
Jun 27 2016 15:47
:)
sorry, i'm half-in-half-out today
Rob Brackett
@Mr0grog
Jun 27 2016 15:48
"scraping," by itself, it a little broad to say "Rx is good/bad for that"
Ross Hinkley
@rosshinkley
Jun 27 2016 15:48
but +1 to everything you've said
rosshinkley @rosshinkley nods
Ross Hinkley
@rosshinkley
Jun 27 2016 15:48
right tool for the job, etc
Mingsterism
@mingsterism
Jun 27 2016 15:48
@Mr0grog thanks Rob. Heard about highlands. Will check that out.
Rob Brackett
@Mr0grog
Jun 27 2016 15:49
alrighty, I gotta run
Mingsterism
@mingsterism
Jun 27 2016 15:50
@rosshinkley @Mr0grog so is Rob going to be joining chats more too.
:)
Ross Hinkley
@rosshinkley
Jun 27 2016 15:50
i hope so! :)
Mingsterism
@mingsterism
Jun 27 2016 15:51
Haha. Take over your position of being the go to guy to answer everyone's questions. :laughing:
Ross Hinkley
@rosshinkley
Jun 27 2016 15:52
i think he tries to pitch in when/where he can, as do i :)
Mingsterism
@mingsterism
Jun 27 2016 15:54
Yeah. I'm actually starting to keep track of the time u guys come online. :laughing:
the catch part
Ross Hinkley
@rosshinkley
Jun 27 2016 16:00
uuhhh, actually, i think the then...catch problem was fixed
hang on
see... rosshinkley/nightmare-examples#27 and segmentio/nightmare#583
as you can see, i'm a bit behind :P
Dmitri
@katabatic
Jun 27 2016 16:02
oh :)
thx
Ross Hinkley
@rosshinkley
Jun 27 2016 16:02
no problem :)
... answering your question a little more directly, were you asking why the .then() was of the form .then(resolve, reject)?
Dmitri
@katabatic
Jun 27 2016 16:05
well I was confused about the discrepancy, and as to which to use
the discrepancy between the example
Ross Hinkley
@rosshinkley
Jun 27 2016 16:05
i think in general you want to use .then().catch()
.then(resolve, reject) is considered an antipattern by some
Dmitri
@katabatic
Jun 27 2016 16:11
ah, thanks
Ross Hinkley
@rosshinkley
Jun 27 2016 16:11
np :)
(man, i cannot type for the life of me today)
coyotte508
@coyotte508
Jun 27 2016 16:30
From the docs:
var nightmare = Nightmare({
  webPreferences: {
    preload: custom-script.js
  }
})
is custom-script.js the path to a file?

I tried it that way, and even after adding

window.__nightmare = {};
__nightmare.ipc = require('ipc');

at the beginning of the file, my nightmare is stuck

do i need to install the 'ipc' module locally as well? (Isn't the preload script something evaluated in the browser?)
Ross Hinkley
@rosshinkley
Jun 27 2016 16:37

is custom-script.js the path to a file?

yes.

do i need to install the 'ipc' module locally as well?

no.

(Isn't the preload script something evaluated in the browser?)

yes, but it runs prior to almost the entirety of the render cycle.

the documentation is a bit misleading on preload scripts
coyotte508
@coyotte508
Jun 27 2016 16:38
alright thanks
i'll test a bit more then and try to get it working
Ross Hinkley
@rosshinkley
Jun 27 2016 16:40
the stock preload hooks up the error forwarding and some other sugar for modal dialogs that you'd have to set up yourself if you're using them
also...
i think that got deprecated in favor of require('electron').ipcRenderer
i don't think require('ipc') will work
Dmitri
@katabatic
Jun 27 2016 16:42
nightmare looks very nice
Ross Hinkley
@rosshinkley
Jun 27 2016 16:43
bah, the documentation didn't get updated when we upgraded to electron 1.x, my apologies @coyotte508
coyotte508
@coyotte508
Jun 27 2016 16:43
thanks :)
Ross Hinkley
@rosshinkley
Jun 27 2016 16:43
thanks for letting me know! :) i'll submit an issue so i don't forget to fix it
(unless you'd like to)
coyotte508
@coyotte508
Jun 27 2016 16:45
I can make an issue but for the PR I need to get it working in my tests first
Ross Hinkley
@rosshinkley
Jun 27 2016 16:46
sorry, issue submitted... segmentio/nightmare#700
lmk when you get it working
coyotte508
@coyotte508
Jun 27 2016 16:50
i got a basic version working, but things like nightmare.on("console", ...) don't seem to forward console.log in the preload script (maybe that's waht you meant earlier)
Ross Hinkley
@rosshinkley
Jun 27 2016 16:51
like this?
coyotte508
@coyotte508
Jun 27 2016 16:51
oh I see
Ross Hinkley
@rosshinkley
Jun 27 2016 16:52
the only drawback to using custom preloads, you don't get any of that for free
coyotte508
@coyotte508
Jun 27 2016 17:22
well
i tried using preloads to cancel images loading (by settings their src to '' after DOM is loaded) to gain time, but it seems that only something like editing the html received before it's processed would help
coyotte508
@coyotte508
Jun 27 2016 17:36
and it's a simple setting to give to webPreferences... facepalm
Ross Hinkley
@rosshinkley
Jun 27 2016 18:05
lol
sorry
i stepped away for a minute
aaand i have to go again, glad you figured it out though :)