开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> Python知识库 -> 爬虫学习——（一）Request库请求网页 -> 正文阅读

[Python知识库]爬虫学习——（一）Request库请求网页

一、requests库常见方式请求网页

1.GET请求

在使用之前，确保安装requests库，可以在任务栏cmd里使用pip3来安装：

pip3 install requests

一、requests库常见方式请求网页

import requests

r = requests.get('https://www.httpbin.org/get')
r = requests.post('https://www.httpbin.org/post')
r = requests.put('https://www.httpbin.org/put')
r = requests.delete('https://www.httpbin.org/delete')
r = requests.patch('https://www.httpbin.org/patch')

1.GET请求

基本实例

import requests

r = requests.get('https://www.httpbin.org/get')
print(r.text)

运行结果：

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "www.httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-62d27f11-52cfa9120c9cb3430ff3acbf"
  }, 
  "origin": "120.238.232.103", 
  "url": "https://www.httpbin.org/get"
}

由上，返回结果包括args，headers(请求头)，IP，URL等信息。如果想要添加两个参数name和age，可以利用参数设置传递该信息，其中参数以字典的形式传入

import requests
data = {
    "name":"LZQ",
    "age" : 22
}
r = requests.get('https://www.httpbin.org/get',params=data)
print(r.text)

运行结果：

{
  "args": {
    "age": "22", 
    "name": "LZQ"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "www.httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-62d28029-58d27395661203591b286a70"
  }, 
  "origin": "120.238.232.103", 
  "url": "https://www.httpbin.org/get?name=LZQ&age=22"
}

添加请求头

import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"}
r = requests.get('https://www.httpbin.org/get',headers = headers)
print(r.text)

运行结果：：

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "www.httpbin.org", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-62d29e95-7f32281c719e7d9b4847c851"
  }, 
  "origin": "120.238.218.103", 
  "url": "https://www.httpbin.org/get"
}

百度网页http请求实战

?下面构建一个最简单的get请求，请求链接为：https://www.baidu.com

import requests
import json
r = requests.get('https://www.baidu.com/')
print(r.status_code)  #状态码
print(r.text)  #响应体
print(r.headers)  #响应头

200
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>???o|??????????°±?￥é</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=???o|???? class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>?°é?</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>?°??</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>è§é￠</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>è′′?§</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>????</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">????</a>');
                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">?′?¤?o§?</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>?3?o???o|</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>????¨???o|???èˉ?</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>?è§?é|</a>&nbsp;?o?ICPèˉ030173?·&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>

{'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Sat, 16 Jul 2022 11:15:27 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:23:55 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}

抓取网页数据

import requests
import re
r = requests.get("https://ssr1.scrape.center/")
pattern = re.compile('<h2.*?>(.*?)</h2>',re.S)
titles = re.findall(pattern,r.text)
print(titles)

运行结果：

['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption', '泰坦尼克号 - Titanic', '罗马假日 - Roman Holiday', '唐伯虎点秋香 - Flirting Scholar', '乱世佳人 - Gone with the Wind', '喜剧之王 - The King of Comedy', '楚门的世界 - The Truman Show', '狮子王 - The Lion King']

上述利用正则表达式抓取该网站的标题，后面会更新学习正则表达式的相关内容。

抓取二进制数据

图片、音频、视频等文件本质上都是由二进制码组成的，有特定的保存格式和对应的解码格式，如果想抓取这些数据，就必须抓取他们的二进制数据。

import requests

r = requests.get("https://ssr1.scrape.center/static/img/logo.png")
print(r.text)
print(r.content)

运行结果部分截图：

?对比response的两个属性，注意到r.text出现乱码，而r.content的前面带有一个b，代表是bytes类型的数据，现将提取出的数据进行保存。

import requests

r = requests.get("https://ssr1.scrape.center/static/img/logo.png")
with open('logo.png','wb') as f:  #第一个参数的文件名称，第二个参数是以二进制写的形式打开文件
    f.write(r.content)

2.POST请求

使用requests实现post请求也十分简单，实例如下：

import requests

data = {
    "name":"LZQ",
    "age" : 22
}
r = requests.post('https://www.httpbin.org/post',params=data)
print(r.text)

运行结果：

{
  "args": {
    "age": "22", 
    "name": "LZQ"
  }, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "0", 
    "Host": "www.httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-62d29f4b-17b7d6b13b6d5c8d4a65f987"
  }, 
  "json": null, 
  "origin": "120.238.218.103", 
  "url": "https://www.httpbin.org/post?name=LZQ&age=22"
}

3.响应

请求发送后会得到响应，可以用来获得状态码、响应头、url等其他信息。

import requests

r = requests.get("https://www.baidu.com")
print(type(r.status_code),r.status_code) #状态码
print(type(r.headers),r.headers)  #响应头
print(type(r.cookies),r.cookies)  #Cookie
print(type(r.url),r.url) 
print(type(r.history),r.history) #历史记录

运行结果：

<class 'int'> 200
<class 'requests.structures.CaseInsensitiveDict'> {'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Sat, 16 Jul 2022 11:26:22 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:23:55 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}
<class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
<class 'str'> 百度一下，你就知道
<class 'list'> []