These are chat archives for rosshinkley/nightmare

14th
Jun 2017
Dale Moore
@zeroEvidence
Jun 14 2017 18:35
Hi all.
I'm using nightmare to scrape data from a webapp that has a lot of content being updated at short and regular intervals, I'm interested in only a small portion of the data that is being updated, is it possible to "listen" / "observe" a dom element for changes then have nightmare to execute a function accordingly?
The only solution I can think of is to have a plugin that accepts css selectors, create a MutationObserver to watch for changes and then to send the data to my application via an AJAX request.
Can anyone offer suggestions? or recommend a better solution?
Rob Brackett
@Mr0grog
Jun 14 2017 20:05
A plugin is almost certainly the best way to accomplish that—I think your idea is on the right track. You could also rig it in a more ad-hoc way:
function scrapeStuff() {
  nightmareInstance
    .evaluate(() => {
      return document.querySelector('#something').textContent;
    })
    .then(text => {
      // do something with the found text
    });
}

nightmareInstance
  .on('console', (type, arg) => {
    if (type === 'log' && arg === '::PAGE_UPDATED::') {
      scrapeStuff();
    }
  })
  .evaluate(() => {
    const observer = new MutationObserver(() => {
      console.log('::PAGE_UPDATED::');
    });
    observer.observe(document.querySelector('#something'), {
      childList: true,
      characterData: true,
      subtree: true
    });
  });
Rob Brackett
@Mr0grog
Jun 14 2017 20:14
Obviously you could include more details as args to the console.log call if you wanted to know more about what changed.
Dale Moore
@zeroEvidence
Jun 14 2017 22:33
Thanks Rob, that's very helpful! Your solution is a better solution, I hadn't considered that I could log the required details and have nightmare listen to the console logs, this solution allows me to access the data without having to break the context. Perfect!! I'm a happy man, thank you again!!