Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Mar 11 16:32
    TASnomad closed #72
  • Jan 31 19:33
    bogdanRada commented #73
  • Jan 31 19:29
    bogdanRada commented #73
  • Jan 29 18:48

    dependabot[bot] on npm_and_yarn

    (compare)

  • Jan 29 18:48

    paps on master

    Bump sshpk from 1.13.0 to 1.16.… Merge pull request #74 from pha… (compare)

  • Jan 29 18:48
    paps closed #74
  • Jan 29 18:45
    dependabot[bot] labeled #74
  • Jan 29 18:45
    dependabot[bot] opened #74
  • Jan 29 18:45

    dependabot[bot] on npm_and_yarn

    Bump sshpk from 1.13.0 to 1.16.… (compare)

  • Jan 29 18:45

    paps on master

    Fix verification for childStdou… Merge pull request #73 from bog… (compare)

  • Jan 29 18:45
    paps closed #73
  • Jan 28 08:07
    bogdanRada commented #73
  • Jan 09 16:12
    serverlessguru commented #16
  • Jan 07 10:12
    bogdanRada synchronize #73
  • Jan 07 10:09
    bogdanRada opened #73
  • Jan 02 11:05
    paps commented #71
  • Dec 27 2019 08:00
    allgrowsm commented #61
  • Dec 27 2019 03:23
    allgrowsm commented #61
  • Dec 26 2019 12:24
    paps commented #61
  • Dec 26 2019 09:48
    allgrowsm commented #61
customxx
@customxx
Hi guys here a REGEX phone number
(?:(?:+?([1-9]|[0-9][0-9]|[0-9][0-9][0-9])\s(?:[.-]\s)?)?(?:(\s([2-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9])\s)|([0-9][1-9]|[0-9]1[02-9]|[2-9][02-8]1|[2-9][02-8][02-9]))\s(?:[.-]\s)?)?([2-9]1[02-9]|[2-9][02-9]1|[2-9][02-9]{2})\s(?:[.-]\s)?([0-9]{4})(?:\s(?:#|x.?|ext.?|extension)\s(\d+))?
and custom script made with base on email extractor
here the example god luck :)
oshjain
@oshjain
Hi there, How can I make nick.js await inside a page evaluate?
Also, I'm unable to debug code inside page.evaluate? How can I do that
charliezone
@charliezone
to solveNoCaptcha I have to pay some external service?
Avi Aialon
@aviaialon
anyone still here?
Martin Barreau
@TASnomad
Yes @aviaialon
Avi Aialon
@aviaialon
Hi Martin, im trying to use nickjs, but i dont get the same results as when i run the same example on nickjs.org - are there any specific specs that was used on the .org site, os & version, chrome os + version etc.?
Martin Barreau
@TASnomad
@aviaialon, if you face an error with a Fatal error from Chrome subprocess while running Nickjs, it could probably come from a bad configuration of your chrome, could you try to run google-chrome -enable-logging=stderr -v1 --headless google.com? (you can the —no-sandbox flag too)
Spencer Brower
@sbrow

Hello, I'm trying to scrape search results page-by-page, but I can't get nickjs to load pages past the first. Here is a code example:

(async () => {
    let results: string[]; // @returns
    let n; // number of pages to scrape.
    for (let i = 0; i < n; i++) {
        if (i > 0) {
            try {
                await tab.untilVisible(opts.next); // "Next Page" Button/link.
                await tab.click(opts.next);
                await tab.untilVisible(opts.previous); // "Previous Page" Button/link
            } catch (e) {
                HandleError(e);
            }
        }
        await tab.untilVisible(opts.result);
        await tab.inject("./node_modules/jquery/dist/jquery.min.js"); // We're going to use jQuery to scrape
        try {
            results = await tab.evaluate(opts, scrapeResult);
        } catch (e) {
            HandleError(e);
        }
        return results;
    }
});

Anyone know the solution? Even if I try waiting for a significant period of time it still doesn't work. Thanks for the help. :)

Martin Barreau
@TASnomad
Hello @sbrow, have you called nick.newTab() & tab.open() before scraping? Do you have any errors when you call the open method?
Spencer Brower
@sbrow
Hey @TASnomad, Thanks for the response. I am calling nick.newTab() and tab.open(), and I get good results from the page. However, after the tab.click() method, I'm not seeing a change in the page, when the element I'm clicking on has a click event.
Spencer Brower
@sbrow
Discovered the problem, user error. The website I was scraping was throwing a popup instead of advancing to the next page.
Ghost
@ghost~5c994fc4d73408ce4fbbcad2
This message was deleted
Payflic Support
@Payflic_Support_twitter
Any ideas on inputting values in forms inside iframes without opening a new tab to access the iframe content?
Martin Barreau
@TASnomad
@Payflic_Support_twitter you can try an evaluate to find the iframe document . You just need to call querySelector(__iframes_selector__).contentDocument to have the iframe dociment & use querySelector calls from the contentDocument to set values in the iframe form
Payflic Support
@Payflic_Support_twitter
Thanks @TASnomad i actually figured this out. For anyone else who is struggling this with this and doesn't want to spend hours digging through google results here's a solution example:
const completeIFrameForm = (arg, cb) => {

    var iframe = document.querySelector("#Iframe_selector").contentWindow

    const nameInput = iframe.document.querySelector("#name").value = arg.name //Selector from within the iFrame


    cb(null, true)


}

const completeForm = await tab.evaluate(completeIFrameForm, {name})
Shripad Deshmukh
@shripadde_twitter
Hi
And can I use this library in php
Avi Aialon
@aviaialon

Hi, im struggling to open a URL with POST. Im using the chrome driver with Google Chrome 62.0.3202.75, in the doc, it seems the options argument allows request method POST and data. for example:

var options = {
                method: 'post',
                data: {
                    test: 1
                }
}
await tab.open(..., options)

But the requests are not coming in as POST (and the data isnt sent as well)

Martin Barreau
@TASnomad
Hello @aviaialon, according to the open method documentation (you can find it here: https://hub.phantombuster.com/reference#nick-open) options argument is ignored when by chrome driver except the CasperJS one. Were you using the CasperJS driver?
Avi Aialon
@aviaialon
@TASnomad I was using the Chrome driver. So theres no way to open a URL with POST when using the Chrome driver?
Martin Barreau
@TASnomad
Yes, it’s not possible with the Chrome driver to open an URL with a POST
công ngô chí
@leehanse_twitter
Hello, when i run nickjs i see error: could not start chrome: Error: spawn google-chrome-beta ENOENT
how to fix it
Martin Barreau
@TASnomad
Hello @leehanse_twitter, you can run this shell command export CHROME_PATH=/path/to/chrome/executable before launching your nickjs script. This shell line will tell to nickjs where is your chrome executable.
AbdesJS
@AbdesJS
Hello guys
do you have any example of code with a POST request?
when I test mine I receive this error:
Something went wrong: TypeError: tab.driver.client.Network.setRequestInterception is not a function
bonjour Martin
@TASnomad
@TASnomad
Sennesi
@Sennesi
Hello. is it possible to run the lib nick asynchronously? to work in several newTab at the same time?
Martin Barreau
@TASnomad

Hello @AbdesJS
setRequestInterception is marked as experimental (and deprecated) on the Chrome DevTools Protocol, it recommended to use Fetch.enable instead (https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable)

You can try to use the following example as a NickJS evaluate to send a POST request:

const sendXHR = (arg, cb) => {
    const doXHR = ({ method, url, headers, data }) => {
        return new Promise((resolve, reject) => {
            let xhr = new XMLHttpRequest()

            xhr.addEventListener("readystatechange", function() {
                if (this.readyState === this.DONE) {
                    resolve(xhr.responseText)
                }
            })

            xhr.onerror = function() {
                reject({ status: this.status, statusText: xhr.statusText })
            }

            xhr.open(method, url)
            if (headers) {
                Object.keys(headers).forEach(el => xhr.setRequestHeader(el, headers[el]))
            }
            xhr.send(data)
        })
    }
    cb(null, doXHR(arg))
}
AbdesJS
@AbdesJS
thanks @TASnomad
PayFlic
@PayFlic_ltd_twitter
Hi, got a really frustrating one here. Im trying to automate a product page and i need to be able to select the size option which is contained in a dropdown list. Any idea how to navigate drop down lists with NICKJS?
<select aria-label="size-dropdown" class="size-dropdown" name="size-dropdown">

<option value="0">Select Size</option>

<option value="3924119">XS</option>

<option value="3924120">S</option>

<option value="3924121">M</option>

<option value="3924122">L</option>

<option value="3924123">XL</option>

<option value="3924124" selected="selected">XXL</option>

<option value="3924125">XXXL</option></select>
Martin Barreau
@TASnomad

Hello @PayFlic_ltd_twitter, you can use the following NickJS evaluate snippet to select a specific option by it’s value:

const selectOptionByContent = (arg, cb) => {
    const optionValue = "XS"
    let options = […document.querySelectorAll("select[name=\"size-dropdown\"] option")]
    options = options.filter(el => el.textContent).filter(el => el.textContent.trim().toLowerCase() === optionValue.toLowerCase())
    if (options.length < 1) {
        return cb(`option ${optionValue} not found`)
    }
    const target = options[0]
    target.selected = true
    return cb(null, true)
}

If you already know the option value tag to target you can simply use in a NickJS evaluate: document.querySelector("select[name=\"size-dropdown\"] option=\"3924119\"").selected = true

PayFlic
@PayFlic_ltd_twitter
@TASnomad brilliant thanks
Vi Nguyen
@ngtrieuvi92

hello all, I'm trying to run chrome without headless mode for debug on my MacOS, but i can not run it, the code return this error

CHROME STDERR: [1129/143352.575017:ERROR:chrome_main_delegate.cc(617)] Web security may only be disabled if '--user-data-dir' is also specified.
CHROME STDOUT: Opening in existing browser session.

Fatal: Chrome subprocess exited with code 0

and this is my config:

const nick = new Nick({
  timeout: 60000,
  headless: false,
  debug: true,
});

can someone help me solve this issues?

Martin Barreau
@TASnomad
Hi @ngtrieuvi92, which NickJS version triggers this fatal error?
Sumate
@sumate-sdk
Why it so hard to find starting point tutorial for novice when they say is so simple to use?
Can anyone share me some proper tutorial just like scrapy?
I use less than one day to deal with scrapy but the simple NickJS T-T
masao soseki
@allgrowsm
@sumate-sdk I guess you did not set up properly. Tutorial is easy enough for everyone.
Hello. Is there any option to select without headless mode? Sometime I need without headless mode. I can just comment out the source code. But, I feel like this is not good when nickjs is updated.
kshilov
@kshilov

Hi Guys!

Jsut start playing with nickjs - looks clean.

Trying to understand - what's the major difference from puppeteer? When to use nickjs rather that puppeteer?