V2EX = way to explore V2EX 是一个关于分享和探索的地方
Sign Up Now For Existing Member Sign In
推荐学习书目 Learn Python the Hard Way Python Sites PyPI - Python Package Index http://diveintopython.org/toc/index.html Pocoo 值得关注的项目 PyPy Celery Jinja2 Read the Docs gevent pyenv virtualenv Stackless Python Beautiful Soup 结巴中文分词 Green Unicorn Sentry Shovel Pyflakes pytest Python 编程 pep8 Checker Styles PEP 8 Google Python Style Guide Code Style from The Hitchhiker's Guide
This topic created in 1935 days ago, the information mentioned may be changed or developed.
有挺多 scarpy 的爬虫，受限于代理数量，没法一次性全部启动。于是想找到一个类似队列，取哪个爬哪个的方法。 目前试过 进程池、scrapy 接口 都不是很理性 进程池的问题:
每个进程消费完第一个爬虫，进程就会关闭
scrapy 接口的问题:
必须按顺序爬，假如其他进程跑完，剩下未跑完的爬虫还是得一个个按顺序跑，浪费时间
以下是进程池的代码
from scrapy.crawler import CrawlerProcess from multiprocessing import Pool def _crawl_main_program(spider, settings): # spider: 爬虫 process = CrawlerProcess(settings) process.crawl(spider) process.start() def _crawl_running(crawl_map: dict, settings: dict, max_processes=5): # crawl_map: scrapy 爬虫映射表 if not crawl_map: raise CrawlError() executor = Pool(processes=max_processes) for domain, spider in crawl_map.items(): executor.apply_async(_crawl_main_program, (spider, settings)) executor.close() executor.join() def core_website_crawl(): _crawl_running(crawl_map=core_spider_domain_map, settings=core_website_crawl_settings) if __name__ == '__main__': core_website_crawl() 想找到一个比较好用的方法
Select Language