Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 21 03:09
    allgrowsm edited #71
  • Nov 21 03:04
    allgrowsm edited #71
  • Nov 21 03:03
    allgrowsm edited #71
  • Nov 21 03:02
    allgrowsm opened #71
  • Nov 01 14:12
    paps commented #63
  • Nov 01 14:10

    paps on master

    0.3.7 (compare)

  • Nov 01 13:58

    paps on master

    update package lock (compare)

  • Nov 01 13:57

    dependabot[bot] on npm_and_yarn

    (compare)

  • Nov 01 13:57

    paps on master

    Bump mime from 1.3.6 to 1.6.0 … Merge pull request #70 from pha… (compare)

  • Nov 01 13:57
    paps closed #70
  • Nov 01 13:57
    dependabot[bot] synchronize #70
  • Nov 01 13:57

    dependabot[bot] on npm_and_yarn

    Bump mime from 1.3.6 to 1.6.0 … (compare)

  • Nov 01 13:57
    dependabot[bot] edited #70
  • Nov 01 13:54
    dependabot[bot] edited #70
  • Nov 01 13:54

    paps on master

    update package lock (compare)

  • Nov 01 13:52
    dependabot[bot] labeled #70
  • Nov 01 13:52
    dependabot[bot] opened #70
  • Nov 01 13:52

    dependabot[bot] on npm_and_yarn

    Bump mime from 1.3.6 to 1.6.0 … (compare)

  • Nov 01 13:51

    dependabot[bot] on npm_and_yarn

    (compare)

  • Nov 01 13:51

    paps on master

    Bump lodash from 4.17.4 to 4.17… Merge pull request #67 from pha… (compare)

Ghost
@ghost~5b1f4bcbd73408ce4f9d0151
The output is next to it. @DrunkenTaco619
DrunkenTaco619
@DrunkenTaco619
is there a way to see the nick() method? i understand it's being used but would like to see the method itself.
Dysproh Sium
@dysproh_gitlab
Headless chrome (running through Nickjs) isn't downloading files (specifically, downloadable .mp3 files). Is there a fix?
jeremiecourlivant
@jeremiecourlivant
Hello. How access to an element in an iframe with Nickjs ?
Ghost
@ghost~5b1f4bcbd73408ce4f9d0151
@jeremiecourlivant I guess you can use evaluate() to get iframe.
customxx
@customxx
Hi guys here a REGEX phone number
(?:(?:+?([1-9]|[0-9][0-9]|[0-9][0-9][0-9])\s(?:[.-]\s)?)?(?:(\s([2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\s)|([0-9][1-9]|[0-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\s(?:[.-]\s)?)?([2-9]1[02-9]|[2-9][02-9]1|[2-9][02-9]{2})\s(?:[.-]\s)?([0-9]{4})(?:\s(?:#|x.?|ext.?|extension)\s(\d+))?
and custom script made with base on email extractor
here the example god luck :)
oshjain
@oshjain
Hi there, How can I make nick.js await inside a page evaluate?
Also, I'm unable to debug code inside page.evaluate? How can I do that
charliezone
@charliezone
to solveNoCaptcha I have to pay some external service?
Avi Aialon
@aviaialon
anyone still here?
Martin Barreau
@TASnomad
Yes @aviaialon
Avi Aialon
@aviaialon
Hi Martin, im trying to use nickjs, but i dont get the same results as when i run the same example on nickjs.org - are there any specific specs that was used on the .org site, os & version, chrome os + version etc.?
Martin Barreau
@TASnomad
@aviaialon, if you face an error with a Fatal error from Chrome subprocess while running Nickjs, it could probably come from a bad configuration of your chrome, could you try to run google-chrome -enable-logging=stderr -v1 --headless google.com? (you can the —no-sandbox flag too)
Spencer Brower
@sbrow

Hello, I'm trying to scrape search results page-by-page, but I can't get nickjs to load pages past the first. Here is a code example:

(async () => {
    let results: string[]; // @returns
    let n; // number of pages to scrape.
    for (let i = 0; i < n; i++) {
        if (i > 0) {
            try {
                await tab.untilVisible(opts.next); // "Next Page" Button/link.
                await tab.click(opts.next);
                await tab.untilVisible(opts.previous); // "Previous Page" Button/link
            } catch (e) {
                HandleError(e);
            }
        }
        await tab.untilVisible(opts.result);
        await tab.inject("./node_modules/jquery/dist/jquery.min.js"); // We're going to use jQuery to scrape
        try {
            results = await tab.evaluate(opts, scrapeResult);
        } catch (e) {
            HandleError(e);
        }
        return results;
    }
});

Anyone know the solution? Even if I try waiting for a significant period of time it still doesn't work. Thanks for the help. :)

Martin Barreau
@TASnomad
Hello @sbrow, have you called nick.newTab() & tab.open() before scraping? Do you have any errors when you call the open method?
Spencer Brower
@sbrow
Hey @TASnomad, Thanks for the response. I am calling nick.newTab() and tab.open(), and I get good results from the page. However, after the tab.click() method, I'm not seeing a change in the page, when the element I'm clicking on has a click event.
Spencer Brower
@sbrow
Discovered the problem, user error. The website I was scraping was throwing a popup instead of advancing to the next page.
Ghost
@ghost~5c994fc4d73408ce4fbbcad2
This message was deleted
Payflic Support
@Payflic_Support_twitter
Any ideas on inputting values in forms inside iframes without opening a new tab to access the iframe content?
Martin Barreau
@TASnomad
@Payflic_Support_twitter you can try an evaluate to find the iframe document . You just need to call querySelector(__iframes_selector__).contentDocument to have the iframe dociment & use querySelector calls from the contentDocument to set values in the iframe form
Payflic Support
@Payflic_Support_twitter
Thanks @TASnomad i actually figured this out. For anyone else who is struggling this with this and doesn't want to spend hours digging through google results here's a solution example:
const completeIFrameForm = (arg, cb) => {

    var iframe = document.querySelector("#Iframe_selector").contentWindow

    const nameInput = iframe.document.querySelector("#name").value = arg.name //Selector from within the iFrame


    cb(null, true)


}

const completeForm = await tab.evaluate(completeIFrameForm, {name})
Shripad Deshmukh
@shripadde_twitter
Hi
And can I use this library in php
Avi Aialon
@aviaialon

Hi, im struggling to open a URL with POST. Im using the chrome driver with Google Chrome 62.0.3202.75, in the doc, it seems the options argument allows request method POST and data. for example:

var options = {
                method: 'post',
                data: {
                    test: 1
                }
}
await tab.open(..., options)

But the requests are not coming in as POST (and the data isnt sent as well)

Martin Barreau
@TASnomad
Hello @aviaialon, according to the open method documentation (you can find it here: https://hub.phantombuster.com/reference#nick-open) options argument is ignored when by chrome driver except the CasperJS one. Were you using the CasperJS driver?
Avi Aialon
@aviaialon
@TASnomad I was using the Chrome driver. So theres no way to open a URL with POST when using the Chrome driver?
Martin Barreau
@TASnomad
Yes, it’s not possible with the Chrome driver to open an URL with a POST
công ngô chí
@leehanse_twitter
Hello, when i run nickjs i see error: could not start chrome: Error: spawn google-chrome-beta ENOENT
how to fix it
Martin Barreau
@TASnomad
Hello @leehanse_twitter, you can run this shell command export CHROME_PATH=/path/to/chrome/executable before launching your nickjs script. This shell line will tell to nickjs where is your chrome executable.
AbdesJS
@AbdesJS
Hello guys
do you have any example of code with a POST request?
when I test mine I receive this error:
Something went wrong: TypeError: tab.driver.client.Network.setRequestInterception is not a function
bonjour Martin
@TASnomad
@TASnomad
Sennesi
@Sennesi
Hello. is it possible to run the lib nick asynchronously? to work in several newTab at the same time?
Martin Barreau
@TASnomad

Hello @AbdesJS
setRequestInterception is marked as experimental (and deprecated) on the Chrome DevTools Protocol, it recommended to use Fetch.enable instead (https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable)

You can try to use the following example as a NickJS evaluate to send a POST request:

const sendXHR = (arg, cb) => {
    const doXHR = ({ method, url, headers, data }) => {
        return new Promise((resolve, reject) => {
            let xhr = new XMLHttpRequest()

            xhr.addEventListener("readystatechange", function() {
                if (this.readyState === this.DONE) {
                    resolve(xhr.responseText)
                }
            })

            xhr.onerror = function() {
                reject({ status: this.status, statusText: xhr.statusText })
            }

            xhr.open(method, url)
            if (headers) {
                Object.keys(headers).forEach(el => xhr.setRequestHeader(el, headers[el]))
            }
            xhr.send(data)
        })
    }
    cb(null, doXHR(arg))
}
AbdesJS
@AbdesJS
thanks @TASnomad
PayFlic
@PayFlic_ltd_twitter
Hi, got a really frustrating one here. Im trying to automate a product page and i need to be able to select the size option which is contained in a dropdown list. Any idea how to navigate drop down lists with NICKJS?
<select aria-label="size-dropdown" class="size-dropdown" name="size-dropdown">

<option value="0">Select Size</option>

<option value="3924119">XS</option>

<option value="3924120">S</option>

<option value="3924121">M</option>

<option value="3924122">L</option>

<option value="3924123">XL</option>

<option value="3924124" selected="selected">XXL</option>

<option value="3924125">XXXL</option></select>
Martin Barreau
@TASnomad

Hello @PayFlic_ltd_twitter, you can use the following NickJS evaluate snippet to select a specific option by it’s value:

const selectOptionByContent = (arg, cb) => {
    const optionValue = "XS"
    let options = […document.querySelectorAll("select[name=\"size-dropdown\"] option")]
    options = options.filter(el => el.textContent).filter(el => el.textContent.trim().toLowerCase() === optionValue.toLowerCase())
    if (options.length < 1) {
        return cb(`option ${optionValue} not found`)
    }
    const target = options[0]
    target.selected = true
    return cb(null, true)
}

If you already know the option value tag to target you can simply use in a NickJS evaluate: document.querySelector("select[name=\"size-dropdown\"] option=\"3924119\"").selected = true

PayFlic
@PayFlic_ltd_twitter
@TASnomad brilliant thanks
Vi Nguyen
@ngtrieuvi92

hello all, I'm trying to run chrome without headless mode for debug on my MacOS, but i can not run it, the code return this error

CHROME STDERR: [1129/143352.575017:ERROR:chrome_main_delegate.cc(617)] Web security may only be disabled if '--user-data-dir' is also specified.
CHROME STDOUT: Opening in existing browser session.

Fatal: Chrome subprocess exited with code 0

and this is my config:

const nick = new Nick({
  timeout: 60000,
  headless: false,
  debug: true,
});

can someone help me solve this issues?

Martin Barreau
@TASnomad
Hi @ngtrieuvi92, which NickJS version triggers this fatal error?