scrapy的大概运行流程

scrapy.cmdline.py

scrapy.utils.project.py

scrapy.settings.__init__.py

scrapy.settings.default_settings.py（部分展示）

分支主题

scrapy.command.crawl.py

scrapy.command.__init__.py

scrapy.cralwer.py

子主题

scrpay.spiders.__init__.py

scrapy.core.engine.py

scrapy.core.downloader.py

scrapy.core.downloader.handlers.__init__.py

自由主题

_add_middleware

DownloaderMiddlewareManager

scrapy.core.scraper.py

scrapy.core.spidermw.py

scrapy.pipelines.__init__.py

scrapy.middleware.py

start_requests

spider

open_spider

from_crawler

open

enqueue_request

_dqpush

scrapy.core.scheduler.py

scrapy.utils.misc.py（创建对象的一个方法）

request_seen

scrapy.duperfilters.py

部分可更改的队列设置

scrapy.reactor.py

scrapy.dupefilter.py

Scraper.open_spider

ExecutionEngine._next_request

ExecutionEngine._needs_backout

ExecutionEngine._next_request_from_scheduler

ExecutionEngine.crawl

ExecutionEngine._schedule_request

scrapy.utils.request.py

ExecutionEngine._download

scrapy.downloader.__init__.py Downloader

scrapy.downloader.middleware.py

scrapy.downloader.handlers.__init__.py    DownloadHandlers

scrapy.downloader.handlers.__init__.py    DownloadHandlers

ExecutionEngine._handle_downloader_output

scrapy.core.scraper.py  Scraper

scrapy.core.scraper.py  Scraper

scrapy.core.spidermw.py  SpiderMiddlewareManager

scrapy.__main__.py