Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 19 17:20
    paps commented #73
  • Jun 19 17:07

    paps on master

    0.3.8 (compare)

  • Jun 19 15:21
    paps commented #80
  • Jun 19 14:52
    msaus opened #80
  • Jun 19 14:30
    dependabot[bot] labeled #79
  • Jun 19 14:30
    dependabot[bot] opened #79
  • Jun 19 14:30
    dependabot[bot] labeled #78
  • Jun 19 14:30

    dependabot[bot] on npm_and_yarn

    Bump extend from 3.0.1 to 3.0.2… (compare)

  • Jun 19 14:30
    dependabot[bot] opened #78
  • Jun 19 14:30

    dependabot[bot] on npm_and_yarn

    Bump stringstream from 0.0.5 to… (compare)

  • Jun 19 14:30

    paps on master

    Add deprecation notice in README fix puppeteer link fix deprecation notice in README and 2 more (compare)

  • Jun 19 14:30
    paps closed #77
  • Jun 17 12:13
    stressGC review_requested #77
  • Jun 17 12:13
    stressGC assigned #77
  • Jun 17 12:13
    stressGC opened #77
  • Jun 17 12:09

    stressGC on deprecation

    remove doc broken link (compare)

  • Jun 17 12:06

    stressGC on deprecation

    fix deprecation notice in README (compare)

  • Jun 17 07:51

    stressGC on deprecation

    fix puppeteer link (compare)

  • Jun 17 07:50

    stressGC on deprecation

    Add deprecation notice in README (compare)

  • Apr 03 13:56
    spearmootz opened #76
James Russo
@jamesrusso
What is the best way to wait for certain text on a page?
Seems like not possible with a Selector?
James Russo
@jamesrusso
I wish I could do a :contains selector..
Martin Tapia
@paps
@jamesrusso you can do a tab.evaluate() containing a setTimeout() recursive loop, waiting for something to appear
@crimsonsoccer55 no, to do that you can use await tab.driver.client.Page.addScriptToEvaluateOnNewDocument, see here https://chromedevtools.github.io/devtools-protocol/tot/Page#method-addScriptToEvaluateOnNewDocument
James Russo
@jamesrusso
Thanks @paps.. I was just able to find some other elements..
James Russo
@jamesrusso
Any want to “wait for next page to load”? I’m running into an issue where I’m clicking on something which is causing a location.href change, and then I’m trying to wait for new elements to load in an evaluate, but the problem is that the evaluate runs on the page prior to it navigating away.
Martin Tapia
@paps
@jamesrusso yeah the evaluate() call will be forcefully terminated when the page changes
You have to return from it (with useful data if needed) and make another evaluate() call just after, when the new page has begun loading
DrunkenTaco619
@DrunkenTaco619
can someone explain to me what this does and why it quit working after i ran it once?
Ghost
@ghost~5b1f4bcbd73408ce4f9d0151
The output is next to it. @DrunkenTaco619
DrunkenTaco619
@DrunkenTaco619
is there a way to see the nick() method? i understand it's being used but would like to see the method itself.
Dysproh Sium
@dysproh_gitlab
Headless chrome (running through Nickjs) isn't downloading files (specifically, downloadable .mp3 files). Is there a fix?
jeremiecourlivant
@jeremiecourlivant
Hello. How access to an element in an iframe with Nickjs ?
Ghost
@ghost~5b1f4bcbd73408ce4f9d0151
@jeremiecourlivant I guess you can use evaluate() to get iframe.
customxx
@customxx
Hi guys here a REGEX phone number
(?:(?:+?([1-9]|[0-9][0-9]|[0-9][0-9][0-9])\s(?:[.-]\s)?)?(?:(\s([2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\s)|([0-9][1-9]|[0-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\s(?:[.-]\s)?)?([2-9]1[02-9]|[2-9][02-9]1|[2-9][02-9]{2})\s(?:[.-]\s)?([0-9]{4})(?:\s(?:#|x.?|ext.?|extension)\s(\d+))?
and custom script made with base on email extractor
here the example god luck :)
oshjain
@oshjain
Hi there, How can I make nick.js await inside a page evaluate?
Also, I'm unable to debug code inside page.evaluate? How can I do that
charliezone
@charliezone
to solveNoCaptcha I have to pay some external service?
Avi Aialon
@aviaialon
anyone still here?
Martin Barreau
@TASnomad
Yes @aviaialon
Avi Aialon
@aviaialon
Hi Martin, im trying to use nickjs, but i dont get the same results as when i run the same example on nickjs.org - are there any specific specs that was used on the .org site, os & version, chrome os + version etc.?
Martin Barreau
@TASnomad
@aviaialon, if you face an error with a Fatal error from Chrome subprocess while running Nickjs, it could probably come from a bad configuration of your chrome, could you try to run google-chrome -enable-logging=stderr -v1 --headless google.com? (you can the —no-sandbox flag too)
Spencer Brower
@sbrow

Hello, I'm trying to scrape search results page-by-page, but I can't get nickjs to load pages past the first. Here is a code example:

(async () => {
    let results: string[]; // @returns
    let n; // number of pages to scrape.
    for (let i = 0; i < n; i++) {
        if (i > 0) {
            try {
                await tab.untilVisible(opts.next); // "Next Page" Button/link.
                await tab.click(opts.next);
                await tab.untilVisible(opts.previous); // "Previous Page" Button/link
            } catch (e) {
                HandleError(e);
            }
        }
        await tab.untilVisible(opts.result);
        await tab.inject("./node_modules/jquery/dist/jquery.min.js"); // We're going to use jQuery to scrape
        try {
            results = await tab.evaluate(opts, scrapeResult);
        } catch (e) {
            HandleError(e);
        }
        return results;
    }
});

Anyone know the solution? Even if I try waiting for a significant period of time it still doesn't work. Thanks for the help. :)

Martin Barreau
@TASnomad
Hello @sbrow, have you called nick.newTab() & tab.open() before scraping? Do you have any errors when you call the open method?
Spencer Brower
@sbrow
Hey @TASnomad, Thanks for the response. I am calling nick.newTab() and tab.open(), and I get good results from the page. However, after the tab.click() method, I'm not seeing a change in the page, when the element I'm clicking on has a click event.
Spencer Brower
@sbrow
Discovered the problem, user error. The website I was scraping was throwing a popup instead of advancing to the next page.
Ghost
@ghost~5c994fc4d73408ce4fbbcad2
This message was deleted
Payflic Support
@Payflic_Support_twitter
Any ideas on inputting values in forms inside iframes without opening a new tab to access the iframe content?
Martin Barreau
@TASnomad
@Payflic_Support_twitter you can try an evaluate to find the iframe document . You just need to call querySelector(__iframes_selector__).contentDocument to have the iframe dociment & use querySelector calls from the contentDocument to set values in the iframe form
Payflic Support
@Payflic_Support_twitter
Thanks @TASnomad i actually figured this out. For anyone else who is struggling this with this and doesn't want to spend hours digging through google results here's a solution example:
const completeIFrameForm = (arg, cb) => {

    var iframe = document.querySelector("#Iframe_selector").contentWindow

    const nameInput = iframe.document.querySelector("#name").value = arg.name //Selector from within the iFrame


    cb(null, true)


}

const completeForm = await tab.evaluate(completeIFrameForm, {name})
Shripad Deshmukh
@shripadde_twitter
Hi
And can I use this library in php
Avi Aialon
@aviaialon

Hi, im struggling to open a URL with POST. Im using the chrome driver with Google Chrome 62.0.3202.75, in the doc, it seems the options argument allows request method POST and data. for example:

var options = {
                method: 'post',
                data: {
                    test: 1
                }
}
await tab.open(..., options)

But the requests are not coming in as POST (and the data isnt sent as well)

Martin Barreau
@TASnomad
Hello @aviaialon, according to the open method documentation (you can find it here: https://hub.phantombuster.com/reference#nick-open) options argument is ignored when by chrome driver except the CasperJS one. Were you using the CasperJS driver?
Avi Aialon
@aviaialon
@TASnomad I was using the Chrome driver. So theres no way to open a URL with POST when using the Chrome driver?
Martin Barreau
@TASnomad
Yes, it’s not possible with the Chrome driver to open an URL with a POST
công ngô chí
@leehanse_twitter
Hello, when i run nickjs i see error: could not start chrome: Error: spawn google-chrome-beta ENOENT
how to fix it
Martin Barreau
@TASnomad
Hello @leehanse_twitter, you can run this shell command export CHROME_PATH=/path/to/chrome/executable before launching your nickjs script. This shell line will tell to nickjs where is your chrome executable.
AbdesJS
@AbdesJS
Hello guys
do you have any example of code with a POST request?
when I test mine I receive this error:
Something went wrong: TypeError: tab.driver.client.Network.setRequestInterception is not a function
bonjour Martin
@TASnomad