Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 19 17:20
    paps commented #73
  • Jun 19 17:07

    paps on master

    0.3.8 (compare)

  • Jun 19 15:21
    paps commented #80
  • Jun 19 14:52
    msaus opened #80
  • Jun 19 14:30
    dependabot[bot] labeled #79
  • Jun 19 14:30
    dependabot[bot] opened #79
  • Jun 19 14:30
    dependabot[bot] labeled #78
  • Jun 19 14:30

    dependabot[bot] on npm_and_yarn

    Bump extend from 3.0.1 to 3.0.2… (compare)

  • Jun 19 14:30
    dependabot[bot] opened #78
  • Jun 19 14:30

    dependabot[bot] on npm_and_yarn

    Bump stringstream from 0.0.5 to… (compare)

  • Jun 19 14:30

    paps on master

    Add deprecation notice in README fix puppeteer link fix deprecation notice in README and 2 more (compare)

  • Jun 19 14:30
    paps closed #77
  • Jun 17 12:13
    stressGC review_requested #77
  • Jun 17 12:13
    stressGC assigned #77
  • Jun 17 12:13
    stressGC opened #77
  • Jun 17 12:09

    stressGC on deprecation

    remove doc broken link (compare)

  • Jun 17 12:06

    stressGC on deprecation

    fix deprecation notice in README (compare)

  • Jun 17 07:51

    stressGC on deprecation

    fix puppeteer link (compare)

  • Jun 17 07:50

    stressGC on deprecation

    Add deprecation notice in README (compare)

  • Apr 03 13:56
    spearmootz opened #76
customxx
@customxx
here the example god luck :)
oshjain
@oshjain
Hi there, How can I make nick.js await inside a page evaluate?
Also, I'm unable to debug code inside page.evaluate? How can I do that
charliezone
@charliezone
to solveNoCaptcha I have to pay some external service?
Avi Aialon
@aviaialon
anyone still here?
Martin Barreau
@TASnomad
Yes @aviaialon
Avi Aialon
@aviaialon
Hi Martin, im trying to use nickjs, but i dont get the same results as when i run the same example on nickjs.org - are there any specific specs that was used on the .org site, os & version, chrome os + version etc.?
Martin Barreau
@TASnomad
@aviaialon, if you face an error with a Fatal error from Chrome subprocess while running Nickjs, it could probably come from a bad configuration of your chrome, could you try to run google-chrome -enable-logging=stderr -v1 --headless google.com? (you can the —no-sandbox flag too)
Spencer Brower
@sbrow

Hello, I'm trying to scrape search results page-by-page, but I can't get nickjs to load pages past the first. Here is a code example:

(async () => {
    let results: string[]; // @returns
    let n; // number of pages to scrape.
    for (let i = 0; i < n; i++) {
        if (i > 0) {
            try {
                await tab.untilVisible(opts.next); // "Next Page" Button/link.
                await tab.click(opts.next);
                await tab.untilVisible(opts.previous); // "Previous Page" Button/link
            } catch (e) {
                HandleError(e);
            }
        }
        await tab.untilVisible(opts.result);
        await tab.inject("./node_modules/jquery/dist/jquery.min.js"); // We're going to use jQuery to scrape
        try {
            results = await tab.evaluate(opts, scrapeResult);
        } catch (e) {
            HandleError(e);
        }
        return results;
    }
});

Anyone know the solution? Even if I try waiting for a significant period of time it still doesn't work. Thanks for the help. :)

Martin Barreau
@TASnomad
Hello @sbrow, have you called nick.newTab() & tab.open() before scraping? Do you have any errors when you call the open method?
Spencer Brower
@sbrow
Hey @TASnomad, Thanks for the response. I am calling nick.newTab() and tab.open(), and I get good results from the page. However, after the tab.click() method, I'm not seeing a change in the page, when the element I'm clicking on has a click event.
Spencer Brower
@sbrow
Discovered the problem, user error. The website I was scraping was throwing a popup instead of advancing to the next page.
Ghost
@ghost~5c994fc4d73408ce4fbbcad2
This message was deleted
Payflic Support
@Payflic_Support_twitter
Any ideas on inputting values in forms inside iframes without opening a new tab to access the iframe content?
Martin Barreau
@TASnomad
@Payflic_Support_twitter you can try an evaluate to find the iframe document . You just need to call querySelector(__iframes_selector__).contentDocument to have the iframe dociment & use querySelector calls from the contentDocument to set values in the iframe form
Payflic Support
@Payflic_Support_twitter
Thanks @TASnomad i actually figured this out. For anyone else who is struggling this with this and doesn't want to spend hours digging through google results here's a solution example:
const completeIFrameForm = (arg, cb) => {

    var iframe = document.querySelector("#Iframe_selector").contentWindow

    const nameInput = iframe.document.querySelector("#name").value = arg.name //Selector from within the iFrame


    cb(null, true)


}

const completeForm = await tab.evaluate(completeIFrameForm, {name})
Shripad Deshmukh
@shripadde_twitter
Hi
And can I use this library in php
Avi Aialon
@aviaialon

Hi, im struggling to open a URL with POST. Im using the chrome driver with Google Chrome 62.0.3202.75, in the doc, it seems the options argument allows request method POST and data. for example:

var options = {
                method: 'post',
                data: {
                    test: 1
                }
}
await tab.open(..., options)

But the requests are not coming in as POST (and the data isnt sent as well)

Martin Barreau
@TASnomad
Hello @aviaialon, according to the open method documentation (you can find it here: https://hub.phantombuster.com/reference#nick-open) options argument is ignored when by chrome driver except the CasperJS one. Were you using the CasperJS driver?
Avi Aialon
@aviaialon
@TASnomad I was using the Chrome driver. So theres no way to open a URL with POST when using the Chrome driver?
Martin Barreau
@TASnomad
Yes, it’s not possible with the Chrome driver to open an URL with a POST
công ngô chí
@leehanse_twitter
Hello, when i run nickjs i see error: could not start chrome: Error: spawn google-chrome-beta ENOENT
how to fix it
Martin Barreau
@TASnomad
Hello @leehanse_twitter, you can run this shell command export CHROME_PATH=/path/to/chrome/executable before launching your nickjs script. This shell line will tell to nickjs where is your chrome executable.
AbdesJS
@AbdesJS
Hello guys
do you have any example of code with a POST request?
when I test mine I receive this error:
Something went wrong: TypeError: tab.driver.client.Network.setRequestInterception is not a function
bonjour Martin
@TASnomad
@TASnomad
Sennesi
@Sennesi
Hello. is it possible to run the lib nick asynchronously? to work in several newTab at the same time?
Martin Barreau
@TASnomad

Hello @AbdesJS
setRequestInterception is marked as experimental (and deprecated) on the Chrome DevTools Protocol, it recommended to use Fetch.enable instead (https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable)

You can try to use the following example as a NickJS evaluate to send a POST request:

const sendXHR = (arg, cb) => {
    const doXHR = ({ method, url, headers, data }) => {
        return new Promise((resolve, reject) => {
            let xhr = new XMLHttpRequest()

            xhr.addEventListener("readystatechange", function() {
                if (this.readyState === this.DONE) {
                    resolve(xhr.responseText)
                }
            })

            xhr.onerror = function() {
                reject({ status: this.status, statusText: xhr.statusText })
            }

            xhr.open(method, url)
            if (headers) {
                Object.keys(headers).forEach(el => xhr.setRequestHeader(el, headers[el]))
            }
            xhr.send(data)
        })
    }
    cb(null, doXHR(arg))
}
AbdesJS
@AbdesJS
thanks @TASnomad
PayFlic
@PayFlic_ltd_twitter
Hi, got a really frustrating one here. Im trying to automate a product page and i need to be able to select the size option which is contained in a dropdown list. Any idea how to navigate drop down lists with NICKJS?
<select aria-label="size-dropdown" class="size-dropdown" name="size-dropdown">

<option value="0">Select Size</option>

<option value="3924119">XS</option>

<option value="3924120">S</option>

<option value="3924121">M</option>

<option value="3924122">L</option>

<option value="3924123">XL</option>

<option value="3924124" selected="selected">XXL</option>

<option value="3924125">XXXL</option></select>
Martin Barreau
@TASnomad

Hello @PayFlic_ltd_twitter, you can use the following NickJS evaluate snippet to select a specific option by it’s value:

const selectOptionByContent = (arg, cb) => {
    const optionValue = "XS"
    let options = […document.querySelectorAll("select[name=\"size-dropdown\"] option")]
    options = options.filter(el => el.textContent).filter(el => el.textContent.trim().toLowerCase() === optionValue.toLowerCase())
    if (options.length < 1) {
        return cb(`option ${optionValue} not found`)
    }
    const target = options[0]
    target.selected = true
    return cb(null, true)
}

If you already know the option value tag to target you can simply use in a NickJS evaluate: document.querySelector("select[name=\"size-dropdown\"] option=\"3924119\"").selected = true

PayFlic
@PayFlic_ltd_twitter
@TASnomad brilliant thanks
Vi Nguyen
@ngtrieuvi92

hello all, I'm trying to run chrome without headless mode for debug on my MacOS, but i can not run it, the code return this error

CHROME STDERR: [1129/143352.575017:ERROR:chrome_main_delegate.cc(617)] Web security may only be disabled if '--user-data-dir' is also specified.
CHROME STDOUT: Opening in existing browser session.

Fatal: Chrome subprocess exited with code 0

and this is my config:

const nick = new Nick({
  timeout: 60000,
  headless: false,
  debug: true,
});

can someone help me solve this issues?

Martin Barreau
@TASnomad
Hi @ngtrieuvi92, which NickJS version triggers this fatal error?
Sumate
@sumate-sdk
Why it so hard to find starting point tutorial for novice when they say is so simple to use?
Can anyone share me some proper tutorial just like scrapy?
I use less than one day to deal with scrapy but the simple NickJS T-T
masao soseki
@allgrowsm
@sumate-sdk I guess you did not set up properly. Tutorial is easy enough for everyone.
Hello. Is there any option to select without headless mode? Sometime I need without headless mode. I can just comment out the source code. But, I feel like this is not good when nickjs is updated.
kshilov
@kshilov

Hi Guys!

Jsut start playing with nickjs - looks clean.

Trying to understand - what's the major difference from puppeteer? When to use nickjs rather that puppeteer?

Ricardo Spear
@spearmootz
is nickjs dead?
Martin Tapia
@paps
NickJS has not received a serious commit for more than a year now. We've marked it as 'deprecated' and archived the repo
NickJS was made before Puppeteer existed and it was very useful at that time. Now we recommend you switch to Puppeteer or Playwright :)
DemGiran
@DemGiran
I LOVED NICK