NickJS was great but it's over now 🙏 We recommend you use Puppeteer or Playwright instead
google-chrome -enable-logging=stderr -v1 --headless google.com? (you can the —no-sandbox flag too)
Hello, I'm trying to scrape search results page-by-page, but I can't get nickjs to load pages past the first. Here is a code example:
(async () => {
let results: string[]; // @returns
let n; // number of pages to scrape.
for (let i = 0; i < n; i++) {
if (i > 0) {
try {
await tab.untilVisible(opts.next); // "Next Page" Button/link.
await tab.click(opts.next);
await tab.untilVisible(opts.previous); // "Previous Page" Button/link
} catch (e) {
HandleError(e);
}
}
await tab.untilVisible(opts.result);
await tab.inject("./node_modules/jquery/dist/jquery.min.js"); // We're going to use jQuery to scrape
try {
results = await tab.evaluate(opts, scrapeResult);
} catch (e) {
HandleError(e);
}
return results;
}
});Anyone know the solution? Even if I try waiting for a significant period of time it still doesn't work. Thanks for the help. :)
const completeIFrameForm = (arg, cb) => {
var iframe = document.querySelector("#Iframe_selector").contentWindow
const nameInput = iframe.document.querySelector("#name").value = arg.name //Selector from within the iFrame
cb(null, true)
}
const completeForm = await tab.evaluate(completeIFrameForm, {name})
Business Names from this url
Hi, im struggling to open a URL with POST. Im using the chrome driver with Google Chrome 62.0.3202.75, in the doc, it seems the options argument allows request method POST and data. for example:
var options = {
method: 'post',
data: {
test: 1
}
}
await tab.open(..., options)But the requests are not coming in as POST (and the data isnt sent as well)
open method documentation (you can find it here: https://hub.phantombuster.com/reference#nick-open) options argument is ignored when by chrome driver except the CasperJS one. Were you using the CasperJS driver?
Something went wrong: TypeError: tab.driver.client.Network.setRequestInterception is not a function
Hello @AbdesJSsetRequestInterception is marked as experimental (and deprecated) on the Chrome DevTools Protocol, it recommended to use Fetch.enable instead (https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable)
You can try to use the following example as a NickJS evaluate to send a POST request:
const sendXHR = (arg, cb) => {
const doXHR = ({ method, url, headers, data }) => {
return new Promise((resolve, reject) => {
let xhr = new XMLHttpRequest()
xhr.addEventListener("readystatechange", function() {
if (this.readyState === this.DONE) {
resolve(xhr.responseText)
}
})
xhr.onerror = function() {
reject({ status: this.status, statusText: xhr.statusText })
}
xhr.open(method, url)
if (headers) {
Object.keys(headers).forEach(el => xhr.setRequestHeader(el, headers[el]))
}
xhr.send(data)
})
}
cb(null, doXHR(arg))
}