These are chat archives for rosshinkley/nightmare

23rd
Jun 2016
Mingsterism
@mingsterism
Jun 23 2016 01:23
@rosshinkley Hi Ross. Is this the right way to structure nightmare code. i have a feeling that pattern of .then.then.then is really simplistic. I can do some refactoring. But any tips on how to improve further. Lack your experience to take it to the next level. :)
.then(function() {
        return nightmare
            .evaluate(nextPage) // CHECK IF NEXT PAGE RETURNS LINK
        })
    .then(function(nextUrl) {
        console.log('Received Next Url......')
        console.log(nextUrl)
        return nightmare
            .goto(nextUrl)
            .evaluate(jobTitles)
    }, function(err) {
        console.log(' Error evaluating next page ');
        console.log(err);
        return nightmare             
            // DO SOMETHING ELSE HERE
    })
    .then(function(t) {
        console.log(t)
        return nightmare 
            // DO SOMETHING ELSE HERE
    })
also, i put all my logic (if statements) in the function. is that good?
and just one other question, I'm not entirely clear on the role of generators. Are they applicable here?
sorry if im asking you too many questions. Please let me know if i am.
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:00
Is there a way to evaluate if an element is available based on the xPath?
Mingsterism
@mingsterism
Jun 23 2016 13:02
might help.
just put it in your evaluate() function
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:09
ill try it, thanks
Mingsterism
@mingsterism
Jun 23 2016 13:18
no probs man. anything let us know
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:31
I'm back with another question. I'm following the example on the github readme that uses mocha for testing. When I run the test I'm getting a timeout failure even after I set timeout to 10 seconds. It did work once, I'm not sure why. Any ideas?
Mingsterism
@mingsterism
Jun 23 2016 13:32
does it load the site?
have you put nightmare = Nightmare{show:true}
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:32
a detail that might be important is when I run without mocha there is a popup, when I run with mocha there isn't a popup. It looks like it tries but doesn't appear
ok let me try that
Mingsterism
@mingsterism
Jun 23 2016 13:32
you can see your electron browser to debug what is happening.
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:34
show:true doesn't work with mocha snippet
Mingsterism
@mingsterism
Jun 23 2016 13:34
try it without mocha first.
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:34
without mocha worked
Mingsterism
@mingsterism
Jun 23 2016 13:35
i see. i cant be of much help. i dont use mocha.
@rosshinkley will know
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:36
alright
do you know how to run tests in other browsers? is it possible?
for example, I want to test functionality using firefox and chrome can I do that?
Mingsterism
@mingsterism
Jun 23 2016 13:41
electron runs on chromium
means google chrome. dont think it can do firefox or others.
but i may be wrong
Alex Bednarczyk
@alexbednarczyk
Jun 23 2016 13:42
alright thanks
Ross Hinkley
@rosshinkley
Jun 23 2016 15:53
@mingsterism re where you put the ifs, what function are you referring? depends on what you're trying to do
broadly speaking, generators (in this context) help you make more synchronous-looking code
nightmare-examples has some examples with vo and co
and... i did a writeup on that...
i've been considering editing that and moving it to nightmare-examples
Mingsterism
@mingsterism
Jun 23 2016 15:56
Thanks Ross. Yeah, those generator examples will be useful.
Ross Hinkley
@rosshinkley
Jun 23 2016 15:56
and, you're not asking too many questions @mingsterism, ask away :)
Mingsterism
@mingsterism
Jun 23 2016 15:56
haha. thanks :)
Ross Hinkley
@rosshinkley
Jun 23 2016 15:57
as for you, @alexbednarczyk, regarding xpath...
@mingsterism was right, you can use document.evaluaste() to run xpath queries
however
all of the nightmare methods that take a selector expect CSS queries

I'm back with another question. I'm following the example on the github readme that uses mocha for testing. When I run the test I'm getting a timeout failure even after I set timeout to 10 seconds. It did work once, I'm not sure why. Any ideas?

Hm, which version of the example? It's entirely possible I screwed it up with the last push

i'll take a look at that as time permits today
Mingsterism
@mingsterism
Jun 23 2016 16:01
@rosshinkley I'm wondering, for large codebase, eg in Nightmare, is there a proper structure. For example, placing all functions in 1 file. Or placing all the actions in 1 file? How do people abstract out the details to make it simple, yet flexible?
Ross Hinkley
@rosshinkley
Jun 23 2016 16:01
that sounds like doneisn't getting called
Mingsterism
@mingsterism
Jun 23 2016 16:01
Also, any examples on linking nightmare to database? is mongoose really useful?
Ross Hinkley
@rosshinkley
Jun 23 2016 16:01
@mingsterism that's ... a big question, and i suspect if you ask ten people, you'll get ten answers
Nightmare is divided by some basic moving parts and sugar to get communication and interop working (eg, nightmare.js) and the actions which rely on those parts (actions.js)
which made sense when the codebase was reasonably small
now, it's probably debatable
but core is pretty well locked down with segmentio/nightmare#425 - adding the ability to bolt on more stuff
meaning the API won't change appreciably, but implementation might
.goto() is a great example of this
Mingsterism
@mingsterism
Jun 23 2016 16:05
@rosshinkley any reference / tutorial /book to learn up more on nightmare structure? I notice the use of folders like src, bin, lib in many libraries.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:06
... although, that said, if i ever carve out a day or two to sit down and focus on it, segmentio/nightmare#593 might change some of that surface
oooohhh
good question, and i don't have a good answer
and my personal opinion is that there isn't a single canonical "right" way
Mingsterism
@mingsterism
Jun 23 2016 16:07
i see. i was browsing and came across this. but didnt know if was applicable to other languages.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:08
my guide has usually been keeping concerns separate
possibly? it's easy to overthink directory structure, especially with nodejs
Mingsterism
@mingsterism
Jun 23 2016 16:09
good point.
in regards to linking nightmare to database, any recommendations?
Ross Hinkley
@rosshinkley
Jun 23 2016 16:10
great question, depends on what you're trying to do
mongo(ose) is nice because it's so easy to set up
but there are a lot of options
Mingsterism
@mingsterism
Jun 23 2016 16:11
how to decide? any reference / tutorial?
Ross Hinkley
@rosshinkley
Jun 23 2016 16:12
uh, in general, i decide on a database based on what else the DB has to interact with, what kind of data structures and relationships you need to have, size of the DB, availability...
if you're doing something just for you, mongodb is probably fine
tingoDB is also kind of interesting
it's kind of the sqlite version of mongo
i'm sure there are better selection criteria out there, and DBs are something that are a bit out of my depth
i've worked with many, but i'm not sure i could hold one up over another as "better", necessarily
i could definitely complain about the quirks of each, though ;)
if you tell me a little more about what you're trying to accomplish, i might be able to point you in a more solid direction
i'm realizing now what i said is probably unhelpful :P
Mingsterism
@mingsterism
Jun 23 2016 16:17
actually Im just looking as a place for storing data. because I'm using nightmare to for web scraping.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:17
do you need to query the data later for statistics?
Mingsterism
@mingsterism
Jun 23 2016 16:17
then would want to run it on a scheduler in a cloud.
yeah.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:18
gotcha.
Mingsterism
@mingsterism
Jun 23 2016 16:18
i was thinking of linking to apache spark.
because data analysis done there.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:19
i see
are you planning on using the spark-mongodb wrapper?
Mingsterism
@mingsterism
Jun 23 2016 16:22
i havent done it before actually. so still exploring and open to options
Ross Hinkley
@rosshinkley
Jun 23 2016 16:22
(i know almost nothing about spark, it's been on my list forever)
Mingsterism
@mingsterism
Jun 23 2016 16:22
just wondered if there were any examples/tutorial.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:22
also depends on what kind of data analysis
spark might be overkill :)
Mingsterism
@mingsterism
Jun 23 2016 16:23
yeah. but my thoughts was spark makes things easier infact, because everything is linked together. the data streaming, analysis, and others
just one platform. also, faster than others like hadoop. and for big data, makes it fast to index and can integrate with search engines like elastic search.
really cool. im just started with the book learning spark on amazon.
just if there were some simple database examples on nightmare docs would be helpful.
if not happy to share mine, as soon as i get some simple one started :)
Ross Hinkley
@rosshinkley
Jun 23 2016 16:26
those kinds of details are usually left up to the developer
if you can use mongoose and nightmare, marrying the two shouldn't be difficult
Mingsterism
@mingsterism
Jun 23 2016 16:29
yeah. probably should give mongoose a try.
thanks for the tips ross.
Ross Hinkley
@rosshinkley
Jun 23 2016 16:29
np
mongoose is almost certainly the easiest i've seen to set up :)
Mingsterism
@mingsterism
Jun 23 2016 16:31
i hope so. haha :)