[Python知识库] python爬取素材公社图片

开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> Python知识库 -> python爬取素材公社图片 -> 正文阅读

[Python知识库]python爬取素材公社图片


import requests
import re
# import urllib
# from urllib import request,error

url = "https://www.tooopen.com/aicdr/1096_1100_1_1.aspx"
def askURL(url):
    header = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
        "Referer": "https://www.tooopen.com/aicdr/1096_1100_1_2.aspx",
        "Cookie": "ASP.NET_SessionId=4dff2ec6-6f07-47cc-8de1-8f3478edf6e8; Hm_lvt_d3ac2f8840ead98242d6205eeff29cb4=1652173245; Hm_lpvt_d3ac2f8840ead98242d6205eeff29cb4=1652173337; RefreshFilter=http://www.tooopen.com/ajax/gethistory?callback=jQuery183003647995761540557_1652173336490&_=1652173336598"
    }
    res = requests.get(url,verify=False,headers=header)
    page_content = res.text
    obj = re.compile(r'<a class="pic".*?<img src="(?P<img>.*?)"',re.S)
    result = obj.finditer(page_content)
    m = 0
    imgs = []
    for it in result:
        imgs.append(it.group('img'))

    ###使用urllib下载图片方式
    # m=0
    # for img in imgs:
    #     urllib.request.urlretrieve(img, "d:/download2/" + str(m) + '.jpg')  # 下载图片到本地
    
    ##使用requests下载图片方式

    for imgurl in imgs:
        imgres = requests.get(imgurl, verify=False, headers=header)
        with open("d:/download2/" + str(m) + '.jpg','wb') as f:
            f.write(imgres.content)
        m += 1

askURL(url)

代码提供了两种下载图片的方式，下载图片中，遇到403错误，加上下面这块代码就可以解决了，注意Referer和Cookie一起加上。

 header = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
        "Referer": "https://www.tooopen.com/aicdr/1096_1100_1_2.aspx",
        "Cookie": "ASP.NET_SessionId=4dff2ec6-6f07-47cc-8de1-8f3478edf6e8; Hm_lvt_d3ac2f8840ead98242d6205eeff29cb4=1652173245; Hm_lpvt_d3ac2f8840ead98242d6205eeff29cb4=1652173337; RefreshFilter=http://www.tooopen.com/ajax/gethistory?callback=jQuery183003647995761540557_1652173336490&_=1652173336598"
    }