[Python知识库]一个完整的scrapy爬取图片示例-及简要知识点

1.创建爬虫项目，在指定文件夹的命令窗口下运行
scrapy startproject cnblogspider
项目结构如下

2.添加item
代码如下

import scrapy


class CnblogspiderItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    url=scrapy.Field()
    cimage_urls=scrapy.Field()
    cimage=scrapy.Field()
    image_paths=scrapy.Field()

3.创建spider,在创建的项目文件夹命令窗口下执行
scrapy genspider cnblog cnblogs.com
产生爬虫
然后重写start_urls和parse方法
其中Request为scrapy自带的一个类，项目中用到直接实例化
全部代码如下

import scrapy
from cnblogspider.items import CnblogspiderItem
from scrapy import Request,Selector


class CnblogSpider(scrapy.Spider):
    name = 'cnblog'
    allowed_domains = ['cnblogs.com']
    # start_urls = ['http://cnblogs.com/qiyeboy/default.html?page=1']
    start_urls = ['http://cnblogs.com/qiyeboy/default.html?page=1']

    def parse(self, response):
        # 实现网页的解析
        # 首先抽取所有的文章
        papers=response.xpath(".//*[@class='day']")
        # 从每篇文章中抽取数据
        for paper in papers:
            url = paper.xpath(".//*[@class='postTitle']/a/@href").extract_first()
            # title = paper.xpath(".//*[@class='postTitle']/a/span/text()").extract()[0]
            # time = paper.xpath(".//*[@class='dayTitle']/a/text()").extract()[0]
            # content = paper.xpath(".//*[@class='postTitle']/a/span/text()").extract()[0]
            # print('%s,%s,%s,%s'%url%title%time%content)
            # print(f'{url}{title}{time}{content}')
            # item=CnblogspiderItem(url=url,title=title,time=time,content=content)
            item = CnblogspiderItem(url=url)
            request=scrapy.Request(url=url,callback=self.parse_body)
            # 将item暂存
            request.meta['item']=item
            yield request
            next_page=Selector(response).re(u'<a href="(\s*)">下一页</a>')
            if next_page:
                yield scrapy.Request(url=next_page[0],callback=self.parse)
    def parse_body(self,response):
        item=response.meta['item']
        body=response.xpath(".//*[@class='postBody']")
        # 提取图片链接
        item['cimage_urls']=body.xpath('.//img//@src').extract()
        yield item

4.构建item pipeline存储下载的图片
因为是存储图片所以要用到MediaPipline中的ImagesPipeline
以及设置settings
4.1 要存储到指定的路径下所以重写get_media_requests(item,info)方法和item_completed(results,items,info)方法
其中get_media_requests方法中要用到item的cimage_urls字段
item_completed方法中要用到item的image_paths字段
具体代码如下：

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
import scrapy
from scrapy.pipelines.images import ImagesPipeline


class MyImagesPiplines(ImagesPipeline):
    def get_media_requests(self, item, info):
        for image_url in item['cimage_urls']:
            yield scrapy.Request(image_url)
    def item_completed(self, results, item, info):
        image_paths=[x['path'] for ok,x in results if ok]
        if not image_paths:
            raise DropItem('item contains no image')
        item['image_paths']=image_paths
        return item

4.2设置settings,
4.2.1 因为不能用scrapy初始的user_agent,所以要激活USER_AGENT，并添加自己浏览器的user_agent
代码如下：

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'

4.2.2 激活Pipeline
代码如下

```python
ITEM_PIPELINES = {
   # 'cnblogspider.pipelines.CnblogspiderPipeline': 300,
    'cnblogspider.pipelines.MyImagesPiplines': 301,
}

4.2.3
其他项目为固定项目，根据实际需要设置即可
其中
IMAGES_STORE为文件存储路径字段
IMAGES_URL_FIELD问文件url所在item字段
IMAGES_RESULT_FIELD为文件结果信息所在item所在字段
IMAGES_EXPIRES为文件过期时间（天）
IMAGES_THUMBS制作图片缩略图，并设置缩略图大小
具体代码如下

IMAGES_STORE='D:\\cnblogs'
IMAGES_URL_FIELD='cimage_urls'
IMAGES_RESULT_FIELD='cimages'
IMAGES_EXPIRES=30
IMAGES_THUMBS={
    'small':(50,50),
    'big':(270,270),
}

5.运行爬虫
在项目所在文件夹命令窗口下运行命令
scrapy crawl cnblog
-