![]() ![]() Jason Schwartzman voices the hapless villain, The Spot This sequel utilized the most animators ever used in any single film Jessica Drew makes her on-screen debut as Spider-Woman Phil Lord and Christopher Miller are among the writers attached to the film If your counter value is always 1, then you are still running synchronously. Then run a thread that prints the counter every second. Each time your crawler finishes a request, decrement the counter. Each time your crawler starts a request increment the counter. To test this, create a global object and store a counter on it. ![]() You may have set the env variables correctly but your crawler is written in a way that is processing requests for urls synchronously instead of submitting them to a queue. This may be because you are missing some setup or configuration that would make it so. Your crawler is not actually multi-threaded.I can't imagine there is much CPU work going on in the crawling process so i doubt it's a GIL issue.Įxcluding GIL as an option there are two possibilities here: It sounds like you've already tried this so this is probably not news to you but make sure you have configured the CONCURRENT_REQUESTS and REACTOR_THREADPOOL_MAXSIZE. To get multi-threaded behavior you need to configure your application differently or write code that makes it behave mulit-threaded. It looks like Scrapy Crawlers themselves are single-threaded. Taking a stab at an answer here with no experience of the libraries.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |