These are chat archives for rosshinkley/nightmare

8th
Jul 2016
Mingsterism
@mingsterism
Jul 08 2016 00:29
@Mr0grog Ah I see. Ok. I'm asking because when I run a single bot which goes to google search and scrapes the news section of various search terms, the results are very unpredictable. And when running multiple bots (just 2-3), the failure rate is even higher. Meaning it fails to scrape The news titles. any thoughts why?
Also, is it considered multiple instances if I do th
Rob Brackett
@Mr0grog
Jul 08 2016 05:19
Hmmm, that sounds odd. There is an issue where two instances can conflict on data storage on disk—have you tried using memory partitions?
If you are using XVFB, I think @rosshinkley found an issue where you also need to make sure dbus is running: https://github.com/segmentio/nightmare/blob/master/test/bb-xvfb#L23-L26
so maybe look into that, too
Mingsterism
@mingsterism
Jul 08 2016 05:27
thanks Rob. will check into that.
btw, can get your help on this code. I cant get it too self.nightmare.end() when the loop fails when count > 3
.evaluate(self.pageTitles, '.r._U6c')
.then(function loop(pt, count=0) {
        if (pt.length == 0 && count < 3) {
            return self.nightmare.refresh()
                .wait(1000)
                .evaluate(self.pageTitles, '.r._U6c')    
                .then((pt1) => {
                    return loop(pt1, count +=1)
                }, (err) => {console.log('inner inner err', err)})
        } else if (pt.length == 0 && count > 3) {
            console.log('the loop failed to resolve. just end nightmare') 
            return self.nightmare.end()    
        } else {
            console.log('All tests passed no need to loop')
            return pt; 
        }
    })
@Mr0grog
my count is incrementing, and the loop runs. But when count > 3 the loop does not end and nightmare just sits there running but hanging.
Rob Brackett
@Mr0grog
Jul 08 2016 06:54
And you're not hitting the 'All tests passed no need to loop' case?
Mingsterism
@mingsterism
Jul 08 2016 06:55
no im not hitting that.
if there is an error. if there is no error, then will not enter the loop
then will return the last else statement
Rob Brackett
@Mr0grog
Jul 08 2016 06:57
well, if you call loop the third time and still have ptl being non-empty
you'd hit that case and never call end()
Mingsterism
@mingsterism
Jul 08 2016 06:58
hmm. what did you mean by that?
Rob Brackett
@Mr0grog
Jul 08 2016 06:58
Anyway, do you ever catch()?
if loop itself throws an exception, you're never handling it
e.g. what if pt is null or undefined?
Mingsterism
@mingsterism
Jul 08 2016 06:59
function loop(pt, count=0) {
        console.log('running outer...')
        console.log(pt.length, '=============')
        console.log(count, '###########')
        if (pt.length == 0 && count < 3) {
            return self.nightmare.refresh()
                .wait(1000)
                .evaluate(self.pageTitles, '.r._U6c')    
                .then((pt1) => {
                    console.log('inner inner')
                    console.log(count,'@@@@@@@22')
                    console.log(pt1)
                    return loop(pt1, count +=1)
                }, (err) => {console.log('inner inner err', err)})
        } else {
            console.log('the loop has ended')
            return pt; 
        }
    })
this is stack trace
$ node play1.js
running outer...
0 '============='
0 '###########'
inner inner
0 '@@@@@@@22'
[]
running outer...
0 '============='
1 '###########'
inner inner
1 '@@@@@@@22'
[]
running outer...
0 '============='
2 '###########'
inner inner
2 '@@@@@@@22'
[]
running outer...
0 '============='
3 '###########'
the loop has ended
[]
it just ends there.
oh wait.
Rob Brackett
@Mr0grog
Jul 08 2016 07:00
that code never calls end()
Mingsterism
@mingsterism
Jul 08 2016 07:00
else {
return self.nightmare.end() // works
yeah. im trying out code variations now
yeah my actual one doesnt.
function loop(pt, count=0) {
        console.log('running outer...')
        console.log(pt.length, '=============')
        console.log(count, '###########')
        if (pt.length == 0 && count < 3) {
            return self.nightmare.refresh()
                .wait(1000)
                .evaluate(self.pageTitles, '.r._U6c')    
                .then((pt1) => {
                    console.log('inner inner')
                    console.log(count,'@@@@@@@22')
                    console.log(pt1)
                    return loop(pt1, count +=1)
                }, (err) => {console.log('inner inner err', err)})
        } else if (pt.length == 0 && count > 3) {
            console.log('the loop failed to resolve. just end nightmare')
            return self.nightmare.end()    
        } else {
            console.log('the loop has ended')
            return pt
        }
    })
its in the else if statement
because i want the last else to return the values. only if there is errors, then just .end() it
my stack trace
$ node play1.js
running outer...
0 '============='
0 '###########'
inner inner
0 '@@@@@@@22'
[]
running outer...
0 '============='
1 '###########'
inner inner
1 '@@@@@@@22'
[]
running outer...
0 '============='
2 '###########'
inner inner
2 '@@@@@@@22'
[]
running outer...
0 '============='
3 '###########'
the loop has ended
[]
Rob Brackett
@Mr0grog
Jul 08 2016 07:04
well, you’re still not ever calling end() in this case
Mingsterism
@mingsterism
Jul 08 2016 07:04
@Mr0grog my god. im so silly. should have done this instead
else if (pt.lengh == 0 && count >= 3 i missed the >= sign.
} else if (pt.length == 0 && count >= 3) {
            console.log('the loop failed to resolve. just end nightmare')
            return self.nightmare.end()    
        }
is that correct?
that case did work.
damn it. so careless. sry to bother you with these bug issues.
Rob Brackett
@Mr0grog
Jul 08 2016 07:06
ok, guess I wasn't sure what you were trying to do
Mingsterism
@mingsterism
Jul 08 2016 07:06
yeah. i realised that my code if statements did not take into account when count == 3. because i didnt do count >= 3. but thanks Rob.
what happened to Ross? he disappeared. I think he has a backlog of questions from people. haha.
Rob Brackett
@Mr0grog
Jul 08 2016 07:10
don't know
Mingsterism
@mingsterism
Jul 08 2016 07:10
btw, a quick question. does nightmare tend to prematurely evaluate functions before a page has loaded? I notice sometimes, if I dont have .wait() statements, then evaluate will return null. is that a common problem?
Rob Brackett
@Mr0grog
Jul 08 2016 07:15
no, it should be waiting for the page load to complete, with one exception
if the load takes more than 30 seconds, but the DOM loads in less than 30 seconds
but either way, the DOM should have loaded for sure
otherwise goto will produce an error
it's possible reload is causing some oddness in that regard, though
not really sure
Mingsterism
@mingsterism
Jul 08 2016 07:19

i see. becuase i keep on getting errors like these

$ node play1.js
error reported ..... Unable to find element by selector: #hdtb-msb div:nth-child(2) a
undefined

eg: if i run an instance 5 times, it will happen once. if i run multiple instances simultaneously, it will happen more frequently. just wondering what is the cause. or is this a google specific issue.

Rob Brackett
@Mr0grog
Jul 08 2016 07:20
Is that trying to find search results?
Mingsterism
@mingsterism
Jul 08 2016 07:20
yes. go to google search. enter input. click search. then go google news. grab titles from first page.
Rob Brackett
@Mr0grog
Jul 08 2016 07:21
These days, google loads the results via AJAX, so there's no page load that Nightmare would be waiting to finish
Mingsterism
@mingsterism
Jul 08 2016 07:21
that specific selector is click google news link
sry dont fully understand that. what did you mean "no page load"
Rob Brackett
@Mr0grog
Jul 08 2016 07:21
better to .type('whatever').click('searchbutton').wait('selector').evaluate('selector')
^ note the wait('selector')
Mingsterism
@mingsterism
Jul 08 2016 07:21
ah i see.
Rob Brackett
@Mr0grog
Jul 08 2016 07:22
When you click the search button on Google
it doesn't load a page
Mingsterism
@mingsterism
Jul 08 2016 07:22
beause of AJAX?
Rob Brackett
@Mr0grog
Jul 08 2016 07:23
right, it loads the results with JavaScript and simply adds them to the page
so there's no way for the browser (in this case, Nightmare) to know that things are "loading" and it should wait
Mingsterism
@mingsterism
Jul 08 2016 07:26
but if it waits for a selector, what happens if the page never loads. is that common with google/AJAX ? nightmare will wait until it timeouts?
Rob Brackett
@Mr0grog
Jul 08 2016 07:27
if the selector never shows up, it'll time out and you'll have an error, which I think is what you'd want
Mingsterism
@mingsterism
Jul 08 2016 07:30
Rob why doesnt this work. i want to refactor the code. but it does not recognize pt
.evaluate(...)
.then(loop(pt, count=0))
my original is this
.evaluate(self.pageTitles, '.r._U6c')
.then(function loop(pt, count=0) {
        if (pt.length == 0 && count < 3) {
            return self.nightmare.refresh()
                .wait(1000)
                .evaluate(self.pageTitles, '.r._U6c')    
                .then((pt1) => {
                    return loop(pt1, count +=1)
                }, (err) => {console.log('inner inner err', err)})
        } else if (pt.length == 0 && count >= 3) {
            console.log('the loop failed to resolve. just end nightmare')
            return self.nightmare.end()    
        } else {
            console.log('the loop has ended')
            return pt
        }
    }, (err) => {console.log('error reported .....', err)})
}
Rob Brackett
@Mr0grog
Jul 08 2016 07:33
what did you change it to?
Mingsterism
@mingsterism
Jul 08 2016 07:34
changed to this.
BaseBot.prototype.run = function (search) {

  const self = this
  const loop = function(pt, count=0) {
        if (pt.length == 0 && count < 3) {
            return self.nightmare.refresh()
                .wait(1000)
                .evaluate(self.pageTitles, '.r._U6c')    
                .then((pt1) => {
                    return loop(pt1, count +=1)
                }, (err) => {console.log('inner inner err', err)})
        } else if (pt.length == 0 && count >= 3) {
            console.log('the loop failed to resolve. just end nightmare')
            return self.nightmare.end()    
        } else {
            console.log('the loop has ended')
            return pt
        }
    }
  return self.nightmare
    //load a url
    .goto('https://google.com')
       .insert('#lst-ib', search)
    .click('input[type="submit"][name="btnK"]') 
    .wait(1000)
    .click('#hdtb-msb div:nth-child(2) a')
    .wait(1000)
    .evaluate(self.pageTitles, '.r._U6c')
    .then((pt) => {
        loop(pt, count=0) 
    }, (err) => {console.log('error reported .....', err)})
}
it did loop in the end, but i ended up getting undefined for my results.
$ node play1.js
the loop has ended
reached the outer function.............
undefined
the loop has ended
reached outer trump function
undefined
the loop has ended
undefined
this was my run code.
var bot3 = new BaseBot('lenny')
bot3.run('donald trump')
    .then((x) => {
        console.log('reached outer trump function');
        console.log(x);
        return bot3.nightmare.end()
    })
Rob Brackett
@Mr0grog
Jul 08 2016 07:36
You need to return the result of calling loop there
Mingsterism
@mingsterism
Jul 08 2016 07:36
AHHH.. thanks :)
didnt see that.
Rob Brackett
@Mr0grog
Jul 08 2016 07:36
but why do you have a function and not just .then(loop, err => console.log('error reported .....', err))
Mingsterism
@mingsterism
Jul 08 2016 07:37
that function is recursive.
if for whatever reason the result is bad, it refreshes the page, and refresh again. for 3 times.
is there a better way to do it??
oh WAIT!. you are right.
it did work.
how come though?? because i didnt call my function parameters. or does .then() take in my arguments in its function?
Rob Brackett
@Mr0grog
Jul 08 2016 07:42
in your original version, you passed the function as the argument to then
in your first attempt to just pass in the loop function, you passed its result instead: then(loop(pt, count))
you need to pass the function itself again
then(loop)
anyway, it's my bedtime now; goodnight
Mingsterism
@mingsterism
Jul 08 2016 07:43
alright. thanks Rob. Appreciate the help :)
Ross Hinkley
@rosshinkley
Jul 08 2016 13:42
@mingsterism, sorry, i have been away from this for a bit... let me catch up :)
Ross Hinkley
@rosshinkley
Jul 08 2016 13:47

if you used arch linux before?

Not in many moons... still having an issue?

why doesnt this code work. It appears the inner function fn does not have access to the outer scope as error is document is not defined

the function to create the function in electron is run on the nightmare side. You might want to consider wrapping all of that in a function.

hey does anyone know if nightmare can run on multiple instances.

Looks like @Mr0grog took care of you here. Thanks, Rob!

@mingsterism lmk if you've got any more questions
@antonellopasella Thanks for the gist, I'll put that on my list for today :)

Hello, is there a way to work with Watir on ruby, like phantomjs?

No, Nightmare doesn't support webdrivers. (... should it?)

^-- @mamluka, should have included the name
@Dranithix Not out of the box. I wrote a couple of plugins for this in the wayback: inline download, download manager
Ross Hinkley
@rosshinkley
Jul 08 2016 13:53

hi guys I noticed that nightmare.useragent () that is set isn't inherited to browser frames that are opened with window.open()? Is there a way to enforce this?

@sammyyu how are you controlling windows that aren't the main window? I suspect the answer is yes, but i'd need more information on how you're doing window management first

... i think that's everyone