@umrashrf I’m not sure out of the box how to get logs from Lua but looking at the code I don’t think it would take a lot of effort to add a log in the lua_runner (I don’t know about getting that type of support added but at least locally in a forked version). What type of output are you looking for? errors? etc. https://github.com/scrapinghub/splash/blob/master/splash/lua_runner.py
@hbakhtiyor In order to solve the capcha I need to send it back to the spider, but to do this I can't just return the value, because this closes the tab. So I had to build a service where I was able to send the image via http with LUA.
After that I got to a part of the page were I needed to load a different frame to query on it, but this isn't supported. You can return the value of the frame back to Python, but for some reason I wasn't able to do so. So I ended up switching to Selenium, BTW Firefox Selenium wasn't able to get to this frame either, but luckly Chrome was. (:
After that I got to a part of the page were I needed to load a different frame to query on it, but this isn't supported. You can return the value of the frame back to Python, but for some reason I wasn't able to do so. So I ended up switching to Selenium, BTW Firefox Selenium wasn't able to get to this frame either, but luckly Chrome was. (:
I’m currently doing the following in my Lua script.
if string.find(splash.args.url, “thepage.html") ~= nil then
assert(splash:runjs(“timeOutCheck = function(){return;}"))
end
Actually, the function checks for the existance of a form. If not there then it uses location.href to redirect the browser.
might work
using regexes or selectors
not sure if there is selectors support in Splash
Hi all, I’ve been working on a drop-in replacement for splash that utilises google chrome, in particular, the devtools api to implement the splash HTTP api. It can be easily deployed as a docker container and the repo is hosted at https://github.com/chuckus/chromewhip. It’s still well in early alpha but the container has working functionality. My motivation was simply to start getting some practice with asyncio. Any suggestions, comments or improvements, please file an issue or pull request :)
Any advice for reducing memory consumption?
I've requested splash:html only, without processing any images.
Somebody there #
splash chrome
The difference between them
I'm using it with the scrapy-splash plugin and it fails randomly while scraping
function main(splash)
splash:go("http://example.com")
splash:wait(0.5)
return {html=splash:html()}
end
Something like this
Greetings all. I'm having an issue with the splash docker container. I'm trying to attach a custom module through Docker's volume mounting option and the documentation says that you should run it with the argument "--lua-sandbox-allowed-modules" but Docker is saying that isn't a valid argument. Any ideas? I've disabled sandbox mode for now using the "--disable-lua-sandbox argument" but I'd like to get the other way working.
greetings, if you open this website https://istoe.com.br/moodys-faz-relatorio-positivo-sobre-corte-no-compulsorio-de-bancos-brasileiros/ you will notice that the spinner in your browser keeps spinning after the page appear to be full rendered (there is a broken link for an image and the page tries to download it for 90 seconds). but the splash gives timeout anyway, is there a way to render as is at the timeout cut point?
