To present myself a bit. I'm from France, i'm learning python since 3 years now. I work on 2 crawlers last year: Sparkler (in java) and scrapy cluster. I watch the code of the main composent of ruia (mazing work btw you rock!). And i think i need to work a bit by doing some spiders and testing things by myself before contribute back to ruia mostly because i still struggling with asyncio. I don't know if you already use/ heard off scrapy cluster, these architecture could interest you i think.
About Ruia, i've been working in asyncio, so i came up with the idea of writing an async web scrapping framwork
This project started six months ago, Ruia is still under developing, so feel free to open issues and pull requests
@howie6879 so I did my first spider without difficulty, just got a bug early on crawl because the website didn't like my concurrency. Bu then everything worked perfectly. I was able to scrape 6000 food recipes in 3 hours (i reduced the concurrency a lots). All thanks to you Sir =)
As I said earlier, you need to consider the limitations of the target site.
Just like ip ua limirations etc..
@howie6879 Would making Requests awaitable would make sense? I have a spider with two parse functions. The first one if for iterating through the website and the second for download some files . The second is the purpose of my crawl, yielding it make iterating a lot before downloading the files. I don't know if it make sense, maybe i don't do things correctly
@misterpilou
I don't quite understand what do you mean, can you post details code?
Request.fetch() is awaitable
Perhaps this function is what you need?
I don't quite understand what do you mean, can you post details code?
Request.fetch() is awaitable
Perhaps this function is what you need?
async def parse(self, response):
pages = ['http://httpbin.org/get?p=1' for i in range(10)]
async for resp in self.multiple_request(pages):
yield self.parse_item(response=resp)
For quick explanation, I've got a: yield Request(url=url, callback=self.parse_page) and hoping that awaiting it would call the parse_page function
@misterpilou
Maybe you should define target_item in your subItem class
Hope this tutorials can help you: https://python-ruia.org/en/quickstart.html
Maybe you should define target_item in your subItem class
Hope this tutorials can help you: https://python-ruia.org/en/quickstart.html