Scrapy – Scrapy shell

注解

使用相对文件路径时，请显式并用 ./ （或） ../ 相关时）。 scrapy shell index.html 不会像人们预期的那样工作（这是设计上的，而不是错误）。

因为 shell 喜欢HTTP URL而不是文件URI，以及 index.html 在句法上类似于 example.com ， shell 会治疗 index.html 作为域名并触发DNS查找错误：：

$ scrapy shell index.html
[ ... scrapy shell starts ... ]
[ ... traceback ... ]
twisted.internet.error.DNSLookupError: DNS lookup failed:
address 'index.html' not found: [Errno -5] No address associated with hostname.

shell 如果文件调用了 index.html 存在于当前目录中。同样，要明确。

可用快捷方式¶

• shelp() -打印有关可用对象和快捷方式列表的帮助

• fetch(url[, redirect=True]) - fetch a new response from the given
  URL and update all related objects accordingly. You can optionaly ask for
  HTTP 3xx redirections to not be followed by passing redirect=False

• fetch(request) -从给定的请求中获取新的响应，并相应地更新所有相关对象。

• view(response) -在本地Web浏览器中打开给定的响应以进行检查。这将增加一个 <base> tag 到响应主体，以便外部链接（如图像和样式表）正确显示。但是请注意，这将在您的计算机中创建一个临时文件，该文件不会自动删除。

可用的 Scrapy 对象¶

Scrapy Shell自动从下载的页面创建一些方便的对象，例如 Response 对象与 Selector 对象（用于HTML和XML内容）。

这些对象是：

• crawler -电流 Crawler 对象。

• spider -已知用于处理URL的蜘蛛，或 Spider 如果没有为当前URL找到spider，则为。

• request -A Request 上次提取的页的对象。您可以使用修改此请求 replace() 或者使用 fetch 捷径。

• response -A Response 包含上次提取的页的对象

• settings - the current Scrapy settings

注解

当从命令行运行Scrapy shell时，记住总是用引号括住url，否则url包含参数（即 & 字符）不起作用。

在Windows上，使用双引号：

scrapy shell "https://scrapy.org" --nolog

[settings]
shell = bpython

scrapy shell <url>

# UNIX-style
scrapy shell ./path/to/file.html
scrapy shell ../other/path/to/file.html
scrapy shell /absolute/path/to/file.html
# File URI
scrapy shell file:///absolute/path/to/file.html

scrapy shell 'https://scrapy.org' --nolog

[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f07395dd690>
[s]   item       {}
[s]   request    <GET https://scrapy.org>
[s]   response   <200 https://scrapy.org/>
[s]   settings   <scrapy.settings.Settings object at 0x7f07395dd710>
[s]   spider     <DefaultSpider 'default' at 0x7f0735891690>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
>>>

>>> response.xpath('//title/text()').get()
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'

>>> fetch("https://old.reddit.com/")

>>> response.xpath('//title/text()').get()
'reddit: the front page of the internet'

>>> request = request.replace(method="POST")

>>> fetch(request)

>>> response.status
404

>>> from pprint import pprint

>>> pprint(response.headers)
{'Accept-Ranges': ['bytes'],
 'Cache-Control': ['max-age=0, must-revalidate'],
 'Content-Type': ['text/html; charset=UTF-8'],
 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
 'Server': ['snooserv'],
 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
                'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
 'Vary': ['accept-encoding'],
 'Via': ['1.1 varnish'],
 'X-Cache': ['MISS'],
 'X-Cache-Hits': ['0'],
 'X-Content-Type-Options': ['nosniff'],
 'X-Frame-Options': ['SAMEORIGIN'],
 'X-Moose': ['majestic'],
 'X-Served-By': ['cache-cdg8730-CDG'],
 'X-Timer': ['S1481214079.394283,VS0,VE159'],
 'X-Ua-Compatible': ['IE=edge'],
 'X-Xss-Protection': ['1; mode=block']}

import scrapy
class MySpider(scrapy.Spider):
name = "myspider"
start_urls = [
"http://example.com",
"http://example.org",
"http://example.net",
]
def parse(self, response):
# We want to inspect one specific response.
if ".org" in response.url:
from scrapy.shell import inspect_response
inspect_response(response, self)
# Rest of parsing code.

2014-01-23 17:48:31-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.com> (referer: None)
2014-01-23 17:48:31-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.org> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x1e16b50>
...
>>> response.url
'http://example.org'

>>> response.xpath('//h1[@class="fn"]')
[]

>>> view(response)
True

>>> ^D
2014-01-23 17:50:03-0400 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://example.net> (referer: None)
...