These are chat archives for rosshinkley/nightmare

15th
Dec 2016
Mike Ackerman
@mackermedia
Dec 15 2016 16:24
blob
:wave: Hello! I've been struggling to get a nightmare script running headlessly on Amazon Linux and think I'm allllmost there. I've got past the Xvfb hurdle, but now I'm seeing an error :point_up:
any ideas ? I've tried adding paths: { appData: '/my/path/' } in the nightmare config, but I'm still seeing that same error
Mike Ackerman
@mackermedia
Dec 15 2016 16:30
I'm starting Xvfb separately and passing in the DISPLAY environment variable to the node/nightmare script
Selwyn
@Siilwyn
Dec 15 2016 18:33
@mackermedia did you try running with xvfb-run?
Mike Ackerman
@mackermedia
Dec 15 2016 18:34
I had previously, then I switched to using the ruby headless gem.
Selwyn
@Siilwyn
Dec 15 2016 18:34
though this does no seem related to the display server
Mike Ackerman
@mackermedia
Dec 15 2016 18:35
Now i'm attempting to start Xvfb independently Xvfb :99 -screen 1200x1500x24 as a separate process and passing in DISPLAY=:99 as an environment variable to my node process
both methods result in the same error
Selwyn
@Siilwyn
Dec 15 2016 18:36
hmm so it's missing resolution info
so xvfb-run npm start does not work?
Mike Ackerman
@mackermedia
Dec 15 2016 18:39
testing again quick...
unfortunately when I run it this way any DEBUG log output seems to get swallowed (likely by xvfb-run doing its own process management?)
my script seems to hang, but I believe that's a separate issue related to segmentio/nightmare#555
Selwyn
@Siilwyn
Dec 15 2016 18:46
So I would first try confirm just using a simple nightmare script that it runs in your environment.
Then make sure it really hangs, can test in xvfb-run with a side effect such as writing to a log file.
Mike Ackerman
@mackermedia
Dec 15 2016 18:55
good idea. will report back
Mike Ackerman
@mackermedia
Dec 15 2016 19:11
(btw I'm triggering this script from a Rails app running Passenger application server)
it runs with the following command: xvfb-run -a --server-args="-screen 0 1000x1500x24" node /var/app/current/app/lib/cnn.js
it appears to be hanging: my Rails application response never finishes and a few Electron and one Xvfb processes are still running
it does create a debug.log file with no contents
any more ideas?
trying running this manually via the command line and not via the Rails app
johnferro
@johnferro
Dec 15 2016 19:14
it might be waiting for the main div, probably case sensitive
<div id="Main" class="Col2-stack" role="content" tabindex="-1">
Mike Ackerman
@mackermedia
Dec 15 2016 19:14
ah okay, I copy/pasted that from nightmare github README. i'll try that
hmm, running with just node app/lib/cnn.js exits properly, but no output in debug.log. maybe that overriding console.log doesn't work as I expected it to
johnferro
@johnferro
Dec 15 2016 19:16
with regards to the Xvfb running separately issue, do you get any logging when you start Xvfb? Something like:
Initializing built-in extension RENDER
Initializing built-in extension RANDR
Initializing built-in extension COMPOSITE
Initializing built-in extension DAMAGE
Mike Ackerman
@mackermedia
Dec 15 2016 19:17
johnferro: yes I do
and I would see some output in that terminal window while the nightmare/electron script was running
johnferro
@johnferro
Dec 15 2016 19:18
and it includes RANDR?
Mike Ackerman
@mackermedia
Dec 15 2016 19:18
yes
johnferro
@johnferro
Dec 15 2016 19:19
interesting, unfortunately besides checking that it works for me just now I have only used xvfb-run so don't have any ideas as to what could be going wrong
Mike Ackerman
@mackermedia
Dec 15 2016 19:20
okay. I appreciate your effort
johnferro
@johnferro
Dec 15 2016 19:23
jumping back to the other issue, did you change main in both spots of the script?
Mike Ackerman
@mackermedia
Dec 15 2016 19:25
okay, I've got it working with xvfb-run
will paste output
so it appears to be working properly when manually running this test script (not via Rails app).
also the lowercase #main appears to be working fine
Mike Ackerman
@mackermedia
Dec 15 2016 19:30
triggering the same command via shelling out from within the Ruby/Rails app causes it to hang
has anyone here had luck triggering a nightmare script when shelling out from Ruby?
johnferro
@johnferro
Dec 15 2016 19:31
that's surprising re: lowercase main working, it should be roughly equivalent to running it in the browser's console where I get:
document.querySelector('#main .searchCenterMiddle li a').href
VM264:1 Uncaught TypeError: Cannot read property 'href' of null(…)
although I get that with uppercase main as well
that script from the readme is probably outdated regarding yahoo's selectors
Mike Ackerman
@mackermedia
Dec 15 2016 19:32
in my browser (chrome) it spits out the correct URL
johnferro
@johnferro
Dec 15 2016 19:34
ah ok
no experience starting it from ruby, sorry
Mike Ackerman
@mackermedia
Dec 15 2016 19:36
seems like that might be the main issue :-/
johnferro
@johnferro
Dec 15 2016 19:37
what are you using to start the process from ruby?
Mike Ackerman
@mackermedia
Dec 15 2016 19:42
command = "DEBUG=* xvfb-run -a --server-args=\"-screen 0 1000x1500x24\" node /var/app/current/app/lib/pdf_charts/cnn.js"
`#{command}`
backticks to shell out
Ross Hinkley
@rosshinkley
Dec 15 2016 19:48
poking a head in: does your script end gracefully?
asking because if not, won't the backtics eagerly await an exit?
(been a while since I've done any meaningful ruby)
Mike Ackerman
@mackermedia
Dec 15 2016 19:50
testing with the cnn script, but in my custom script I've got a bunch of catches and they attempt to process.exit(1)
but you're right, ruby will wait for the process to exit
so running with the test cnn script, via the command line on my server with DEBUG=* xvfb-run -a --server-args="-screen 0 1000x1500x24" node /var/app/current/app/lib/pdf_charts/cnn.js it exits fine with a status code of 0
when I trigger it via Ruby, it appears like the process never exits (ruby is sitting waiting endlessly)
does that help clarify?
Ross Hinkley
@rosshinkley
Dec 15 2016 19:51
yyyeah, hm
johnferro
@johnferro
Dec 15 2016 19:53
what nightmare and electron versions are you on?
Mike Ackerman
@mackermedia
Dec 15 2016 19:53
interesting: if I run the command with the test cnn script from my Rails console (rails c) it exits properly
if it gets triggered via the running application (behind Passenger app server) it hangs
nightmare version = 2.8.1, electron version = 1.4.4
Ross Hinkley
@rosshinkley
Dec 15 2016 19:58
weird
I'd expect rails console to behave the same
Mike Ackerman
@mackermedia
Dec 15 2016 19:58
same. i'm very confused
Ross Hinkley
@rosshinkley
Dec 15 2016 19:59
silly question: do they both run under the same user? in your passenger app, can you run ,
which xvfb-run
Mike Ackerman
@mackermedia
Dec 15 2016 20:00
ah, good call. they don't. i'll try switching to the app user and running in rails console there
Ross Hinkley
@rosshinkley
Dec 15 2016 20:00
ah, okay
that may explain the difference then :)
Mike Ackerman
@mackermedia
Dec 15 2016 20:01
although it looks like Passenger is running as root
and the user I was trying on is less priveleged
Ross Hinkley
@rosshinkley
Dec 15 2016 20:02
the alias might be different though
johnferro
@johnferro
Dec 15 2016 20:03
guessing by the ec2 + directory structure combination but are you on elastic beanstalk?
in which case I believe the passenger app is run by a "webapp" user
Ross Hinkley
@rosshinkley
Dec 15 2016 20:03
thought I'd float the idea
Mike Ackerman
@mackermedia
Dec 15 2016 20:04
johnferro: that is correct
we are on elastic beanstalk
application directories are created as webapp user, but the Passenger process is run as root
okay, yep in the Passenger config it specifies user webapp
trying to figure out how to switch to the webapp user. first try it tells me account is currently not available
Mike Ackerman
@mackermedia
Dec 15 2016 20:12
got it. no shell configured
Ross Hinkley
@rosshinkley
Dec 15 2016 20:13
ha! good
Mike Ackerman
@mackermedia
Dec 15 2016 20:13
ah ha!
full circle
Ross Hinkley
@rosshinkley
Dec 15 2016 20:14
that... seems like a permission issue
on the face of it, sorry if I'm rehashing, I'm late to the party
Mike Ackerman
@mackermedia
Dec 15 2016 20:14
no worries. all ideas are welcome
i was attempting to figure out what nightmare/electron thinks is the configured appData path earlier with no success
I'll try again manually setting it to a directory owned by the webapp user
also, it's interesting that when this happens it hangs indefinitely :-/
I'd be open to attempting a PR to make it fail more gracefully if someone could provide some direction
Ross Hinkley
@rosshinkley
Dec 15 2016 20:18
hm
Mike Ackerman
@mackermedia
Dec 15 2016 20:18
so this appears to be an actual issue. I tried configuring nightmare with var nightmare = Nightmare({ show: true, paths: { appData: '/var/app/current/tmp/' } }); and I get the same error
I verified that that directory is owned by the current user
and has write permissions
Ross Hinkley
@rosshinkley
Dec 15 2016 20:23
regarding your first question
i think you can get that info out of electron after it starts
you'd have to use .action() and define that behavior yourself, though
nightmare shouldn't hang if electron crashes... that's a problem :P
i don't have the source in front of me rrright this second
Mike Ackerman
@mackermedia
Dec 15 2016 20:26
sorry, what was that referring to? (the .action() getting info out of electron)
Ross Hinkley
@rosshinkley
Dec 15 2016 20:27
i was attempting to figure out what nightmare/electron thinks is the configured appData path earlier with no success
Mike Ackerman
@mackermedia
Dec 15 2016 20:27
ah okay, thanks
Ross Hinkley
@rosshinkley
Dec 15 2016 20:28
no problem :)
back to the crash: i am pretty certain that if electron dies, nightmare also dies... there's a test for it at the very top of the unit test battery
johnferro
@johnferro
Dec 15 2016 20:29
this may be unrelated but can you check which version of node you're running?
Mike Ackerman
@mackermedia
Dec 15 2016 20:29
node version = v6.9.2
johnferro
@johnferro
Dec 15 2016 20:30
that should be fine then at least!
Ross Hinkley
@rosshinkley
Dec 15 2016 20:30
makes me wonder if you're in an edge case or if that error handling doesn't do what we think :)
(good idea, though, @johnferro, worth checking)
Mike Ackerman
@mackermedia
Dec 15 2016 20:32
I think once we figure out the appData path / permission issue we'd be able to reliably recreate the issue and know if it's worth fixing
Ross Hinkley
@rosshinkley
Dec 15 2016 20:32
:+1:
Ross Hinkley
@rosshinkley
Dec 15 2016 20:38
do you have a fork of nightmare set up?
Mike Ackerman
@mackermedia
Dec 15 2016 20:39
nope
I guess that'd be one way...
Ross Hinkley
@rosshinkley
Dec 15 2016 20:39
fork + symlink, then you can do whatever you want locally :)
(you could also directly patch the package installed in node_modules, i suppose)
(i like fork+symlink because then you also win git, and i have a habit of shooting myself in the foot)
(git eases that pain tremendously)
Mike Ackerman
@mackermedia
Dec 15 2016 20:42
good suggestions
Mike Ackerman
@mackermedia
Dec 15 2016 21:27
update: it looks like manually setting paths: { appData: '/my/directory/' } isn't respected or has a bug
creating the default ~/.config/ directory for the webapp user is making it execute properly
Ross Hinkley
@rosshinkley
Dec 15 2016 21:29
i am somehow unsurprised
Mike Ackerman
@mackermedia
Dec 15 2016 21:38
I'll attempt to create an issue
btw: Thanks for all your hard work on this library. It's great to see another option in the headless browser space :)
Ross Hinkley
@rosshinkley
Dec 15 2016 21:49
ha, i can't take hardly any credit, plenty have contributed
i just wish i had more time :P
Mike Ackerman
@mackermedia
Dec 15 2016 21:49
the eternal struggle
Ross Hinkley
@rosshinkley
Dec 15 2016 21:50
or didn't need to sleep
or eat
;)
Mike Ackerman
@mackermedia
Dec 15 2016 22:01
haha so true. thanks a ton for your help today. fwiw: now I'm fighting something very similar to segmentio/nightmare#555
johnferro
@johnferro
Dec 15 2016 22:19
do you have code you can post?
Mike Ackerman
@mackermedia
Dec 15 2016 22:22
sure. we're attempting to do this synchronously using reduce + promises:
charts.reduce((promise, chart) => {
    return promise.then(() => {
      return page
        .click(chart.tab)
        .wait(chart.selector)
        .hideHeaders()
        .disableD3Animations()
        .evaluate((chart) => {
          var element = document.querySelector(chart.selector)
          element.scrollIntoView(true)
          var rect = element.getBoundingClientRect()

          chart.rect = {
            x:      Math.round(rect.left),
            y:      Math.round(rect.top),
            width:  Math.round(rect.width),
            height: Math.round(rect.height)
          }

          return chart
        }, chart)
        .then((chart) => {
          return page.screenshot(`${saveToDirectory}/${chart.name}.png`, chart.rect)
        }, handleError)
        .catch(handleError)
    })
  }, Promise.resolve()).then(() => {
      page.end().then(() => page.halt(), handleError)
  }, handleError).catch(handleError)
it's inconsistently hanging
if we do just one it seems to finish, adding more than one makes it hang on the last one at:
App 8897 stderr: Thu, 15 Dec 2016 22:25:22 GMT nightmare queueing action "screenshot"
App 8897 stderr: Thu, 15 Dec 2016 22:25:22 GMT nightmare running
App 8897 stderr: Thu, 15 Dec 2016 22:25:22 GMT nightmare:actions .screenshot()
App 8897 stderr: Thu, 15 Dec 2016 22:25:22 GMT nightmare:log subscribing to browser window frames
App 8897 stderr: Thu, 15 Dec 2016 22:25:22 GMT nightmare:log altering page to force rendering
what's odd is that it always hangs on the last one. doesn't matter if it's looping over 2 vs. 6
Mike Ackerman
@mackermedia
Dec 15 2016 22:27
was thinking about trying to convert it to use vo and the generator function format to see if it acts any differently
johnferro
@johnferro
Dec 15 2016 22:31
what promise library are you using?
Mike Ackerman
@mackermedia
Dec 15 2016 22:34
johnferro
@johnferro
Dec 15 2016 22:48
Hmmm it might have to do with the fact that both page.end() and page.halt() close the electron process. Not sure how halt reacts if it tries to kill an already closed process.
Mike Ackerman
@mackermedia
Dec 15 2016 22:52
johnferro: tried changing that line to just page.end() and no difference :-/ . Good thought though
johnferro
@johnferro
Dec 15 2016 22:58
How about returning it like:
}, Promise.resolve()).then(() => {
      return page.end()
  }).then(() => {
   console.log('nightmare ended')
  }).catch(handleError)
Mike Ackerman
@mackermedia
Dec 15 2016 23:03
still hanging
johnferro
@johnferro
Dec 15 2016 23:04
And you said it works with a single item? Even if that is in an array/still using the reduce?
Mike Ackerman
@mackermedia
Dec 15 2016 23:04
correct
trying again with the modifications you suggested
yep, it works with just one
i don't see the output from console.log('nightmare ended') possibly due to xvfb-run swallowing it, but the DEBUG=* logs show:
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare:log subscribing to browser window frames
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare:log altering page to force rendering
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare:log unsubscribing from browser window frames
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare:actions .screenshot() captured with length 4815
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare running
App 13102 stderr: Thu, 15 Dec 2016 23:07:12 GMT nightmare electron child process exited with code 0: success!
johnferro
@johnferro
Dec 15 2016 23:11
Can you check if the electron process is still running or its just the node process that hangs?
Mike Ackerman
@mackermedia
Dec 15 2016 23:14
looks like electron is hanging:
[ec2-user@ip-172-31-57-156 current]$ ps aux | grep electron
webapp   14393 12.0  6.2 1154956 105668 ?      Sl   23:13   0:02 /var/app/current/node_modules/electron/dist/electron /var/app/current/node_modules/nightmare/lib/runner.js {"switches":{"ignore-certificate-errors":true},"loadTimeout":30000,"dock":false}
webapp   14396  0.7  1.6 402944 28620 ?        S    23:13   0:00 /var/app/current/node_modules/electron/dist/electron --type=zygote --no-sandbox
webapp   14410  0.9  1.9 443288 33640 ?        Sl   23:13   0:00 /var/app/current/node_modules/electron/dist/electron --type=gpu-process --channel=14393.0.40773127 --mojo-application-channel-token=8554C7F3A70A2848854210CACA277ACD --no-sandbox --window-depth=24 --x11-visual-id=33 --supports-dual-gpus=false --gpu-driver-bug-workarounds=5,18,56 --gpu-vendor-id=0x0000 --gpu-device-id=0x0000 --gpu-driver-vendor --gpu-driver-version --gpu-driver-date --v8-natives-passed-by-fd --v8-snapshot-passed-by-fd
webapp   14417 54.4  8.5 1175660 145428 ?      Sl   23:13   0:10 /var/app/current/node_modules/electron/dist/electron --type=renderer --no-sandbox --primordial-pipe-token=DB2C6DF66EF54762DAB926D467486A11 --lang=en-US --node-integration=false --preload=/var/app/current/node_modules/nightmare/lib/preload.js --hidden-page -
ec2-user 14495  0.0  0.1 110464  2044 pts/0    S+   23:14   0:00 grep --color=auto electron
btw I love the syntax highlighting in gitter :)
johnferro
@johnferro
Dec 15 2016 23:19
haha yeah its pretty nice
when you were trying to do all 6, was it successfully getting 5 screenshots and then hanging on the last one? or did it still only get the first?
Mike Ackerman
@mackermedia
Dec 15 2016 23:21
when doing all 6, it would successfully do the first 5 and hang on the 6th
seems like whatever the number is > 1, it hangs on the last one