@rosshinkley hey ross. I'm having trouble ending nightmare in a catch statement. I tried many different ways, but it does not seem to end.
any thoughts?
function runScript(url) {
nightmare
.goto(url)
.evaluate(function)
.then(function)
.catch(function(err) {
console.error(err) // <<< This gets called.
return nightmare.end() // <<< this does not work.
})
.then()
.catch(function(err){}) // <<< this does not get called.
}
runScript('http://www.faaaaacebook.com') // wrong url
@mingsterism Two things:
- That last catch() shouldn’t get called, so, technically, everything sounds a-ok there. When you handle a promise error via
catch()or the second argument tothen(), that resolves things in the promise chain so they are no longer in an error state (if you throw an error or return a promise that rejects fromcatch(), though, that will put the promise chain back in to an error state).
It might help to think of it like synchronous code. This:
.then(function(result) {
// do something A
})
.catch(function(error) {
// handle error 1
})
.then(function(result) {
// do something B
})
.catch(function(error) {
// handle error 2
})try {
try {
// do something A
}
catch(error) {
// handle error 1
}
// do something B
}
catch(error) {
// handle error 2
}
Two catch calls in a row will never handle the same error unless the first one re-rejects or throws the error it received.
\2. Are you sure the
nightmare.end() call is not working, or was that just confusion from the above issue? Everything you're doing looks correct there.
If you have nightmare set to show the window, does it close?
@Mr0grog hi Rob. thanks for the clear explaination. i didnt know that the second
.then() function was nested as per your try catch code.
does it mean that if i have 5
.then() means there are 5 nested levels?
Yeah, it's much better to think of things that are in sequence for promises as acting like nesting for synchronous code
should my `return nightmare.end() be in the catch statement?
yeah, that's totally fine
BUT
if you want to make sure you actually end the nightmare instance, you need it in your
then also
because only
then OR catch will run
(assuming you don't also have an error in your
then, of course)
you mean
.evaluate(function) // << error here
.then(function()) // << does not get called
.catch( return nightmare.end())
.then(return nightmare.end())
hmmm, well, that will call
end() more than once
I mean:
.evaluate(something) // maybe error, maybe not
.then(function(result) {
// do what you need with result, then:
return nightmare.end();
})
.catch(function() {
return nightmare.end()
})
oh. but i only wanted to end nightmare if there was error though. if there is no error, i want it to continue.
oh, then don't put it in the
then :)
so just catch?
yeah
I thought you were trying to clean up when you were all done
yeah, but now im testing first with error situation.
then later will place it in end.
:thumbsup:
@Mr0grog wondering could you give me feedback on my code.
what do you think of it
its a very rudementary code. how can i structure it for scale, readability, tdd etc... any thoughts appreciated.
also , the problem i mentioned earlier was in line 58. the
ERRRRRRRRRRRRRRR is called. but nightmare instance does not close.
oh, that's probably because
nightmare is not a local variable there
instead of actually calling
nightmare.end() it is probably throwing an error
line 57 gets called
you need
self.nightmare.end()
ah i see.
my electron window closes but on my console, it does not go to commandline. i must
ctrl-c. does it mean nightmare is still running?
rosshinkley @rosshinkley pokes head in
@mingsterism you're not closing the mongoose connection
probably why your application isn't ending
sorry to butt in :)
no worries Ross. always appreciate your help. :)
any thoughts in regards to how the code can be improved?
so can go in the right direction to improve
hm, i still think
getTitlesAndNextUrl shouldn't be exposed on an instance
yes. sry. will have to get that fixed. but other improvements.
ah, that's fine
beyond that... i don't want to muddy up your process too much
ah. thanks. but was thinking of future scripts though.
you might want to take a look at how
nightmare-swiftly works, and how it uses .use()to create reusable action chains
alright.
(it's a little out of date)
(but it also seems to be in the same spirit of what you're doing with respect to scraping/automation)
haha. yeah. i see last commit was 2 years ago :smile:
cool.
.then(function(v) {
console.log(v.nextUrl, '===========++++++')
Promise.all(v.titles.map(function(title) {
new self.model({job: title}).save((err, result) => {
if (err) console.error(err);
console.log(result);
})
}))
return {
nextUrl: v.nextUrl
}
}).then(function(v) {
console.log(v.nextUrl, '===========++++++')
return Promise.all(v.titles.map(function(title) {
new self.model({job: title}).save()
})).then(function() {
return {nextUrl: v.nextUrl}
});
})
Since you are already catching and logging errors in the next step
you could also just
return v.nextUrl instead of an object with only that in it return {nextUrl: v.nextUrl}
rosshinkley @rosshinkley nods
(Or, if you don't mean to wait for the saves to complete, there's no reason to use
Promise.all there at all)
@Mr0grog wondering is there a difference apart from readability in regards to nesting a
.then() inside the outer .then(function(v)
yes and no…
no in regards to?
if going for readability, (and reusability!) I think the bigger thing is to package that up in a function that you can give a descriptive name to:
function saveTitles(data) {
return Promise.all(data.titles.map(function(title) {
new self.model({job: title}).save()
}))
}
// later...
.then(saveTitles)
or later:
.then(function(data) {
return saveTitles(data).then(Promise.resolve(data.nextUrl));
})
but one way or the other, you kind of need to do that if you want to wait for the saves to complete
interesting. yeah it makes sense to make functions from that.
the real idea with promises is actually that you shouldn't be calling
then() much
you should just be passing the promises around
but when promises got added to JS, they didn't come with the usual types of helpers that the languages they came from had for that
so ¯_(ツ)_/¯ you kind of have to do more yourself
haha. i guess so. thanks for that advise. it will really help me structure my code better.
wondering what do you think of reactive programming
is it a complement or alternative to promises / generators?
rsjs
rxjs
Other super minor nits:
- Get those
require()calls out of the constructor - Because Nightmare has support for newer APIs,
Array.prototype.slice.call(document.querySelectorAll(...))can just beArray.from(document.querySelectorAll())
oh, reactive is very nice
streams are great
I consider them a complement to promises
Streams are like the array version of promises
Much less convenient if you are only trying to get one value (promises are better for that), but very nice otherwise
would be like having:
function sum(a, b) {
return [a + b];
}
But for lists of things or endless series of things, like events, which Rx* is mostly focused on, I think it's pretty awesome
Very similar to promises, in my opinion
is streams part of reactive? are they similar to observables?
observables are streams
just under a different name with more useful tools
but the basic concept is exactly the same
in the case of web scraping tasks, is reactive useful?
There is also http://highlandjs.org, which is a similar idea, but more narrowly-focused
in the case of web scraping tasks, is reactive useful?
Maybe. I don't think it's utility is particular to scraping
hello
if your use case is something like "scraping a long list of URLs" or "URLs will regularly come in and we need to scrape them as they arrive" it definitely would be
hullo!
greetings :)
:)
@Mr0grog or if you're doing a polling mechanism looking for changes
^ that's a good one!
lots of great use cases for Rx/Rx-esque tools
:)
sorry, i'm half-in-half-out today
"scraping," by itself, it a little broad to say "Rx is good/bad for that"
but +1 to everything you've said
rosshinkley @rosshinkley nods
right tool for the job, etc
@Mr0grog thanks Rob. Heard about highlands. Will check that out.
alrighty, I gotta run
@rosshinkley @Mr0grog so is Rob going to be joining chats more too.
:)
i hope so! :)
Haha. Take over your position of being the go to guy to answer everyone's questions. :laughing:
i think he tries to pitch in when/where he can, as do i :)
Yeah. I'm actually starting to keep track of the time u guys come online. :laughing:
the catch part
uuhhh, actually, i think the
then...catch problem was fixed
hang on
see... rosshinkley/nightmare-examples#27 and segmentio/nightmare#583
as you can see, i'm a bit behind :P
oh :)
thx
no problem :)
... answering your question a little more directly, were you asking why the
.then() was of the form .then(resolve, reject)?
well I was confused about the discrepancy, and as to which to use
the discrepancy between the example
i think in general you want to use
.then().catch()
.then(resolve, reject) is considered an antipattern by some
ah, thanks
np :)
(man, i cannot type for the life of me today)
From the docs:
var nightmare = Nightmare({
webPreferences: {
preload: custom-script.js
}
})
is custom-script.js the path to a file?
I tried it that way, and even after adding
window.__nightmare = {};
__nightmare.ipc = require('ipc');at the beginning of the file, my nightmare is stuck
do i need to install the
'ipc' module locally as well? (Isn't the preload script something evaluated in the browser?)
is custom-script.js the path to a file?
yes.
do i need to install the
'ipc'module locally as well?
no.
(Isn't the preload script something evaluated in the browser?)
yes, but it runs prior to almost the entirety of the render cycle.
the documentation is a bit misleading on
preload scripts
alright thanks
i'll test a bit more then and try to get it working
the stock
preload hooks up the error forwarding and some other sugar for modal dialogs that you'd have to set up yourself if you're using them
also...
i think that got deprecated in favor of
require('electron').ipcRenderer
i don't think
require('ipc') will work
nightmare looks very nice
bah, the documentation didn't get updated when we upgraded to electron 1.x, my apologies @coyotte508
thanks :)
thanks for letting me know! :) i'll submit an issue so i don't forget to fix it
(unless you'd like to)
I can make an issue but for the PR I need to get it working in my tests first
sorry, issue submitted... segmentio/nightmare#700
lmk when you get it working
i got a basic version working, but things like
nightmare.on("console", ...) don't seem to forward console.log in the preload script (maybe that's waht you meant earlier)
like this?
oh I see
the only drawback to using custom preloads, you don't get any of that for free
well
i tried using preloads to cancel images loading (by settings their
src to '' after DOM is loaded) to gain time, but it seems that only something like editing the html received before it's processed would help
and it's a simple setting to give to
webPreferences... facepalm
lol
sorry
i stepped away for a minute
aaand i have to go again, glad you figured it out though :)