These are chat archives for rosshinkley/nightmare

10th
Feb 2017
Tiago Loriato
@tloriato
Feb 10 2017 13:56
wish i could help you :/
looking for help with a screenshot problem also
perharps is there someone here that could assist me?
Tiago Loriato
@tloriato
Feb 10 2017 14:04
I opened an issue in the hopes of getting help, if anyone wants to see it: https://github.com/segmentio/nightmare/issues/999#issuecomment-278948616
tl, dr: code is sending the screenshot from the previous run instead of the current one
johnferro
@johnferro
Feb 10 2017 14:39
@jonbr Are you getting a particular error with that code? That looks like the correct way to instantiate a nightmare instance. However, the window itself won't open until you perform an action like using goto. I would try seeing if you could get it working with this example and then go from there: https://github.com/segmentio/nightmare/#examples
johnferro
@johnferro
Feb 10 2017 14:57
@tloriato Screenshots definitely can take awhile depending on the size of the page. Can you change the file name for each run to something unique to that run?
Tiago Loriato
@tloriato
Feb 10 2017 14:59
@johnferro i suppose i can work this out even if i have to hash the arguments, but wouldn't this cause an error when i tried to upload the file to slack and the screenshot hasn't been saved yet?
Tiago Loriato
@tloriato
Feb 10 2017 15:07
@johnferro i don't think that the problem is that it takes time to save the screenshot, but rather that they are only saved after the .end()
I used sleep with 10 seconds to wait for the screenshot and still didn't work, i don't think that if the computer was saving it, it would take more then 10seconds?
yep, increased to 25seconds and still nothing
i'm so close to achieve what i wanted...can someone help me?
Tiago Loriato
@tloriato
Feb 10 2017 15:21
maybe @rosshinkley could shed some light here?
johnferro
@johnferro
Feb 10 2017 15:28
@tloriato It definitely shouldn't require waiting til end. Try intermediary then calls between each screenshot. That should kick off any queued actions.
Tiago Loriato
@tloriato
Feb 10 2017 15:29
@johnferro can i hijack you 2minutos via private?
how could i use this?
.screnshoot()
.then(function(var){

})
what is var?
johnferro
@johnferro
Feb 10 2017 15:32
The result of any queue operation. Not sure what it would be/might not be relevant if you're saving the screenshot. If you weren't saving the screenshot though it would return an image buffer of the screenshot data
Tiago Loriato
@tloriato
Feb 10 2017 15:34
would it be faster to get the buffer than to wait saving the image?
johnferro
@johnferro
Feb 10 2017 15:36
I'm not sure which would be faster. Anecdotally, I've found saving the image to be a bit easier to work with but it probably depends on your specific use case
Tiago Loriato
@tloriato
Feb 10 2017 15:37
hm
johnferro
@johnferro
Feb 10 2017 15:37
I would try first just leaving what you had but using then to kick off the actions early, otherwise it would be waiting for end like you were experiencing
Tiago Loriato
@tloriato
Feb 10 2017 15:37
i can't seem to run the example
i tried this just to get the feel of the thing:
    .screnshot()
    .then(function(title){
        console.log(title);
    });
    .wait(30000)
but the .wait is red now: [js]Declaration or statement expected
johnferro
@johnferro
Feb 10 2017 15:40
The wait function needs to be called off of the nightmare instance. So in the then you could do do something like return nightmare.wait(30000); but you couldn't just call wait on then
Tiago Loriato
@tloriato
Feb 10 2017 15:41
hm
could i just type nightmare.wait(30000) below?
johnferro
@johnferro
Feb 10 2017 15:48
Not if you want it to explicitly wait after the screenshot. It needs to be part of the same promise chain for that
Tiago Loriato
@tloriato
Feb 10 2017 16:04
john, i did the code as shown below:
    .screenshot()
    .then(function(buffer){
        console.log(buffer);
    });
and it prints the buffer as desired
however it does so just after the end, i think
actually, let me check this right
thanks
Ross Hinkley
@rosshinkley
Feb 10 2017 16:06
sorry... let me catch up to you
Tiago Loriato
@tloriato
Feb 10 2017 16:07
sure! thanks
Ross Hinkley
@rosshinkley
Feb 10 2017 16:07
no problem :)
okay
i think i'm caught up, but i apologize if i retread ground you and @johnferro have already covered
Tiago Loriato
@tloriato
Feb 10 2017 16:10
no, that's okay!
i've been stuck for a few days in this already, i'm up for anything
Ross Hinkley
@rosshinkley
Feb 10 2017 16:10
there are two problems here i see nearly immediately
Tiago Loriato
@tloriato
Feb 10 2017 16:11
in the issue I provide my full code if it's helpful
Ross Hinkley
@rosshinkley
Feb 10 2017 16:12
first, the stuff put into the gh issue... none of that will run until .end() is called
which i think is causing you grief
Tiago Loriato
@tloriato
Feb 10 2017 16:13
hm... what do you mean? everything seems to run alright, just that problem with the screenshot...
the bot reads the messages in slack and only starts the nightmare when requested
Ross Hinkley
@rosshinkley
Feb 10 2017 16:14
so, as i read your issue, you want to get the captcha, post to slack, wait for user feedback, use the same instance to do something else based on feedback, then end nightmare
sound right?
Tiago Loriato
@tloriato
Feb 10 2017 16:15
yes!
Ross Hinkley
@rosshinkley
Feb 10 2017 16:16
semi-pseudocode:
 nightmare
    .goto("https://www.receita.fazenda.gov.br/Aplicacoes/SSL/ATCTA/CPF/ConsultaSituacao/ConsultaPublica.asp")
    .type("#txtCPF", cpf)
    .type("#txtDataNascimento", data)
    .screenshot("./step1.png")
    .click("#id_submit")
    .screenshot("./step2.png")
    .wait("#idAntiRobo")
    .screenshot("./captcha.png", { x: 284, y: 317, width: 185, height: 56 })
    .then(()=>{
      //post to slack
      //wait for slack to call back
      //use feedback to fill in captcha (i'm guessing here)
    })
    .then(() => {
      return nightmare.end().then();
    })
Tiago Loriato
@tloriato
Feb 10 2017 16:16
wow...
Ross Hinkley
@rosshinkley
Feb 10 2017 16:16
you've got ... 80% of the pieces
Tiago Loriato
@tloriato
Feb 10 2017 16:17
(i'm guessing here)
haha you guessed right
Ross Hinkley
@rosshinkley
Feb 10 2017 16:17
you can get the captcha and post to slack
which is most of the battle :)
and i'm assuming you've got the stuff written to fill the captcha
and ... continue
(or whatever the next action is)
it's the middle part
Tiago Loriato
@tloriato
Feb 10 2017 16:18
i'm gonna do it rightnow
get back to you in 5, ok?
thank you so much man
Ross Hinkley
@rosshinkley
Feb 10 2017 16:18
hang on
i think i see how to do it
Tiago Loriato
@tloriato
Feb 10 2017 16:18
ok
yeay
Ross Hinkley
@rosshinkley
Feb 10 2017 16:18
in that .then(), you should be able to wait on and pick up the bot calling back
yes?
Tiago Loriato
@tloriato
Feb 10 2017 16:19
yes
Ross Hinkley
@rosshinkley
Feb 10 2017 16:19
yeah
so you're actually much closer than i thought :)
sorry
i wanted to be sure you weren't going to have a hard time getting data back from slack
go forth!
:)
Tiago Loriato
@tloriato
Feb 10 2017 16:20
thanks!
get back to you in 5
Ross Hinkley
@rosshinkley
Feb 10 2017 16:20
sure thing
Tiago Loriato
@tloriato
Feb 10 2017 16:28
hey @rosshinkley can you give me a hand?
Ross Hinkley
@rosshinkley
Feb 10 2017 16:28
shoot
Tiago Loriato
@tloriato
Feb 10 2017 16:30

i created a global variable called captcha, and created a conditional to the bot listen if it's called with the word captcha in it.
the idea is to call it in this format: @bot captcha JNqP23

  else if (msg.text.includes("<@U3ZJRMQF2>") && msg.text.includes("captcha")){
      var arraystring = msg.text.split(/\s+/);
      captcha = arraystring[2];
  }

however I don't know how can I wait for the user input? I tried to do a while(captcha == null){} so the code could wait, but obviously it froze the program

    .then(()=>{
        uploadCaptcha(channel);
        while(captcha == null){
            //loop while waiting for captcha to have a value
        }
        nightmare.type("#txtTexto_captcha_serpro_gov_br", capchta)
        .screenshot("./teste.png");
    })
do you have any ideas?
i think that i can do it with an arbitrary sleep of X seconds, but maybe you have a smarter way
Ross Hinkley
@rosshinkley
Feb 10 2017 16:31
hm, in this case...
could you use the event for the captcha callback to execute more nightmare stuff?
so, chain 1 would end with uploadCaptcha(channel);
then inside of bot.receive(function(){ ... or whatever the syntax is
use chain 2 to do the nightmare.type( ...
Tiago Loriato
@tloriato
Feb 10 2017 16:34
that is clever
haha
thanks!
Ross Hinkley
@rosshinkley
Feb 10 2017 16:36
no problem :)
Tiago Loriato
@tloriato
Feb 10 2017 16:40
ross, i had to move var nightmare to a global variable so i could acess it using the second function
but it didn't work, so I changed show to true
so I could see what was happening
i think that after the first function it closes the browser, so the second function does nothing
Ross Hinkley
@rosshinkley
Feb 10 2017 16:41
do you have a .end() call?
in the first chain?
Tiago Loriato
@tloriato
Feb 10 2017 16:41
nightmare
    .goto(
      "https://www.receita.fazenda.gov.br/Aplicacoes/SSL/ATCTA/CPF/ConsultaSituacao/ConsultaPublica.asp"
    )
    .type("#txtCPF", cpf)
    .type("#txtDataNascimento", data)
    .click("#id_submit")
    .wait("#idAntiRobo")
    .screenshot("./captcha.png", { x: 284, y: 317, width: 185, height: 56 })
    .then(() => {
      uploadCaptcha(channel);
    })
    .then(() => {
      return nightmare.end().then();
    });
}
yes
Ross Hinkley
@rosshinkley
Feb 10 2017 16:41
yeah, don't end it there
put the end in the slack callback
Tiago Loriato
@tloriato
Feb 10 2017 16:42
oh, ok
so i remove this:
.then(() => {
return nightmare.end().then();
});
Ross Hinkley
@rosshinkley
Feb 10 2017 16:42
(that's kind of sloppy, but it'll work for now)
yes
end nightmare only if you mean it ;)
Tiago Loriato
@tloriato
Feb 10 2017 16:43
on the second chain, i just use .end(); or do I copy this code entirely?
Ross Hinkley
@rosshinkley
Feb 10 2017 16:43
you'll still need another .then()
i think
i'd have to see it together :P
Tiago Loriato
@tloriato
Feb 10 2017 16:44
ok, i'll will wrap it up!
alright, finished
var Nightmare = require("nightmare");
var request = require("request");
var sleep = require("sleep");
var fileSystem = require("fs");
var slack = require("slack");
var env = require("node-env-file");
var path = require("path");
var bot = slack.rtm.client();

// load environment variables
env(path.join(__dirname, ".env"));

// grab the auth token
var token = process.env.SLACK_TOKEN;

//Nightmare
  var nightmare = Nightmare({
    switches: {
      "ignore-certificate-errors": true
    },
    show: true
  });


// Wallbo User ID
//<@U3ZJRMQF2>

// handler for messages

//Expected Format:
//@wallbo cpf 111.111.111-11 01/01/1111
bot.message(msg => {
  if (msg.text.includes("<@U3ZJRMQF2>") && !msg.text.includes("captcha")) {
    var arraystring = msg.text.split(/\s+/);
    console.log(arraystring[2]);
    console.log(arraystring[3]);
    var channel = msg.channel;
    StartBrowser(channel, arraystring[2], arraystring[3]);
    var text = "pong! pong!";
    slack.chat.postMessage({ token, channel, text }, console.log);
  } else if (
    msg.text.includes("<@U3ZJRMQF2>") && msg.text.includes("captcha")
  ) {
    var channel = msg.channel;
    var arraystring = msg.text.split(/\s+/);
    var captcha = arraystring[2];
    ContinueBrowser(channel, captcha);
  }
});

// start listening
bot.listen({ token });

//Nightmare
function StartBrowser(channel, cpf, data) {
  nightmare
    .goto(
      "https://www.receita.fazenda.gov.br/Aplicacoes/SSL/ATCTA/CPF/ConsultaSituacao/ConsultaPublica.asp"
    )
    .type("#txtCPF", cpf)
    .type("#txtDataNascimento", data)
    .click("#id_submit")
    .wait("#idAntiRobo")
    .screenshot("./captcha.png", { x: 284, y: 317, width: 185, height: 56 })
    .then(() => {
      uploadCaptcha(channel);
    })
}

function uploadCaptcha(channel) {
  request.post(
    {
      url: "https://slack.com/api/files.upload",
      formData: {
        token: token,
        title: "Captcha",
        filename: "captcha.png",
        filetype: "auto",
        channels: channel,
        file: fileSystem.createReadStream("captcha.png")
      }
    },
    function(err, response) {
      console.log(JSON.parse(response.body));
    }
  );
  return true;
}

function ContinueBrowser(channel, captcha) {
  nightmare
    .type("#txtTexto_captcha_serpro_gov_br", captcha)
    .click("#btnAR2")
    .screenshot("./results.png")
    .then(()=>{
        sendResults(channel);
    })
    .then(() => {
      return nightmare.end().then();
    });
}

function uploadCaptcha(channel) {
  request.post(
    {
      url: "https://slack.com/api/files.upload",
      formData: {
        token: token,
        title: "Results",
        filename: "results.png",
        filetype: "auto",
        channels: channel,
        file: fileSystem.createReadStream("results.png")
      }
    },
    function(err, response) {
      console.log(JSON.parse(response.body));
    }
  );
  return true;
}
thank you @rosshinkley! the whole code is in there and it's working
oops, the last function should be called sendResults (classic ctrl+c cltr+v) error
Ross Hinkley
@rosshinkley
Feb 10 2017 16:49
hm
yeah
also
Tiago Loriato
@tloriato
Feb 10 2017 16:49
i will refactorate it better
but i didn't want to hold you longer
how can i extract the text from a div? i couldn't find on the readme.md from the git
Ross Hinkley
@rosshinkley
Feb 10 2017 16:49
oh, y'know, nevermind
i see what you did here
that'll work just peachy
extract text?
you'll likely need to use .evaluate()
.evaluate(() => document.querySelector('#mydivid').innerText) for example
Tiago Loriato
@tloriato
Feb 10 2017 16:50
right!
thank you so much ross!
Ross Hinkley
@rosshinkley
Feb 10 2017 16:51
no problem
Tiago Loriato
@tloriato
Feb 10 2017 16:51
i will make sure to pass along that kindness when i have the chance
Ross Hinkley
@rosshinkley
Feb 10 2017 16:51
:) that's all i ask
Tiago Loriato
@tloriato
Feb 10 2017 17:00
ross, quick help ahha
how to simulate enter?
i think that .click is not working on the captcha pop-up
Ross Hinkley
@rosshinkley
Feb 10 2017 17:00
oh man, i can't remember if you can send accelerators or not
Tiago Loriato
@tloriato
Feb 10 2017 17:01
.type('body', '\u000d')
acording to the readme.me
just found
Ross Hinkley
@rosshinkley
Feb 10 2017 17:01
i was about to say, i thought you could send the keycode
and that would Just Work :tm:
i vaguely remember having issues on that
but.. it's been long enough since i've had to deal with it that i can't remember
Tiago Loriato
@tloriato
Feb 10 2017 17:02
it doesn't work :(
i can't really graps what's going on
so i don't know how you could help me haha
Ross Hinkley
@rosshinkley
Feb 10 2017 17:03
you could also try to send an enter with .evaluate()
Tiago Loriato
@tloriato
Feb 10 2017 17:04
.evaluate(function (){
//javascript command to click a button
})
right?
Ross Hinkley
@rosshinkley
Feb 10 2017 17:04
if .click() doesn't work, you could try that
you may also want to have a look at....
Tiago Loriato
@tloriato
Feb 10 2017 17:06
i will try that
file:///home/tloriato/Documents/wallbo%20v2/step2.png
ops
do you think that could help?
This message was deleted
maybe the problem it's because it's some form of pop-up?
Ross Hinkley
@rosshinkley
Feb 10 2017 17:07
i doubt it
it looks like a regular css flyover
you may have to get creative with finding the "OK" button, though
Tiago Loriato
@tloriato
Feb 10 2017 17:09
<input id="btnAR2" class="clsButton" type="submit" tabindex="20" value="OK" onclick="javascript: Consultar();"/>
wouldn't .click("#btnAR2") work?
it's funny because nightmare types the captcha right, and then closes, but still take the screenshot as if the cancel button was pressed and not the confirm one
function ContinueBrowser(channel, captcha) {
  nightmare
    .insert("#txtTexto_captcha_serpro_gov_br", captcha)
    .evaluate(function (){

    })
    .click("#btnAR2")
    .screenshot("./results.png")
    .then(()=>{
        sendResults(channel);
    })
    .then(() => {
      return nightmare.end().then();
    });
}
Ross Hinkley
@rosshinkley
Feb 10 2017 17:11
yeah, taht ought to work
although judging from your source there
after the evaluate function, could you just call Consultar() from within evaluate? ;)
Tiago Loriato
@tloriato
Feb 10 2017 17:13
.evaluate(function (){
Consultar();
})
this? :D
Ross Hinkley
@rosshinkley
Feb 10 2017 17:13
eyah
it's worth a try
otherwise, you may want to try real-mouse
Tiago Loriato
@tloriato
Feb 10 2017 17:14
yeah, i will try the real mouse, it didn't work
thanks
IT'S ALIVEEEE
THANK YOU
Ross Hinkley
@rosshinkley
Feb 10 2017 17:17
:)
Tiago Loriato
@tloriato
Feb 10 2017 17:23
ross haha
sorry
(again)
Ross Hinkley
@rosshinkley
Feb 10 2017 17:23
what's up
no worries
Tiago Loriato
@tloriato
Feb 10 2017 17:24

i tried

.evaluate(function (){
        return console.log(document.getElementById('mainComp').innerHTML);
    })

just to try and extract the info from the page just to check what i got

what i would got*
the output was
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Cannot read property 'in
nerHTML' of null
Ross Hinkley
@rosshinkley
Feb 10 2017 17:26
you'll probably want to split it up
like...
.evaluate(() => {
  var element = document.getElementById('mainComp');
  if(element){
    return element.innerHTML;
  } else { 
    return 'mainComp not found'
  }
})
.then(( innerHTML) => console.log(innerHTML))
totally from the hip
Tiago Loriato
@tloriato
Feb 10 2017 17:28
you are a fucking rockstar man
thank you so much
Ross Hinkley
@rosshinkley
Feb 10 2017 17:28
chances are mainComp doesn't exist
Tiago Loriato
@tloriato
Feb 10 2017 17:28
promise to let you work now
Ross Hinkley
@rosshinkley
Feb 10 2017 17:28
eh, i'm not doing anything particularly pressing :)
it's friday after all
Tiago Loriato
@tloriato
Feb 10 2017 17:30
/home/tloriato/Documents/wallbo v2/index.js:108
    .screenshot("./results.png")
     ^

TypeError: nightmare.insert(...).realClick(...).evaluate(...).then(...).screenshot is not a function
function ContinueBrowser(channel, captcha) {
  nightmare
    .insert("#txtTexto_captcha_serpro_gov_br", captcha)
    .realClick("#btnAR2")
    .evaluate(() => {
        var element = document.getElementById('mainComp');
        if(element){
            return element.innerHTML;
        } else { 
            return 'mainComp not found'
        }
        })
    .then(( innerHTML) => console.log(innerHTML))
    .screenshot("./results.png")
    .then(()=>{
        sendResults(channel);
    })
    .then(() => {
      return nightmare.end().then();
    });
}
Ross Hinkley
@rosshinkley
Feb 10 2017 17:31
yyyeah, once you call .then() you end up with a promise, not a nightmare instance
Tiago Loriato
@tloriato
Feb 10 2017 17:31
i think i have to read up alot about promises and then's. I'm really far behind :worried:
Ross Hinkley
@rosshinkley
Feb 10 2017 17:31
you'll want to do
Tiago Loriato
@tloriato
Feb 10 2017 17:31
yeah
just wanted to wrap this up, just missing hah
just put nightmare in front of screenshot then
Ross Hinkley
@rosshinkley
Feb 10 2017 17:32
...
.then((innerHTML) => {
  console.log(innerHTML);
  return nightmare.foo().bar().whatever();
})
.then((resultFromLastChain) => {
})
Tiago Loriato
@tloriato
Feb 10 2017 17:33
right, thanks. that's all for real haha
Ross Hinkley
@rosshinkley
Feb 10 2017 17:33
:)
Tiago Loriato
@tloriato
Feb 10 2017 17:33
if you feel like give me a btc address so i can buy you a coffe :)
Ross Hinkley
@rosshinkley
Feb 10 2017 17:33
lol
Tiago Loriato
@tloriato
Feb 10 2017 17:33
coffee