目录
在使用之前，确保安装requests库，可以在任务栏cmd里使用pip3来安装：
pip3 install requests一、requests库常见方式请求网页
import requests
r = requests.get('https://www.httpbin.org/get')
r = requests.post('https://www.httpbin.org/post')
r = requests.put('https://www.httpbin.org/put')
r = requests.delete('https://www.httpbin.org/delete')
r = requests.patch('https://www.httpbin.org/patch')1.GET请求
-
基本实例
import requests
r = requests.get('https://www.httpbin.org/get')
print(r.text)运行结果：
{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Host": "www.httpbin.org",
"User-Agent": "python-requests/2.27.1",
"X-Amzn-Trace-Id": "Root=1-62d27f11-52cfa9120c9cb3430ff3acbf"
},
"origin": "120.238.232.103",
"url": "https://www.httpbin.org/get"
}
由上，返回结果包括args，headers(请求头)，IP，URL等信息。如果想要添加两个参数name和age，可以利用参数设置传递该信息，其中参数以字典的形式传入
import requests
data = {
"name":"LZQ",
"age" : 22
}
r = requests.get('https://www.httpbin.org/get',params=data)
print(r.text)运行结果：
{
"args": {
"age": "22",
"name": "LZQ"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Host": "www.httpbin.org",
"User-Agent": "python-requests/2.27.1",
"X-Amzn-Trace-Id": "Root=1-62d28029-58d27395661203591b286a70"
},
"origin": "120.238.232.103",
"url": "https://www.httpbin.org/get?name=LZQ&age=22"
}
-
添加请求头
import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"}
r = requests.get('https://www.httpbin.org/get',headers = headers)
print(r.text)运行结果：：
{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Host": "www.httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-62d29e95-7f32281c719e7d9b4847c851"
},
"origin": "120.238.218.103",
"url": "https://www.httpbin.org/get"
}
-
百度网页http请求实战
?下面构建一个最简单的get请求，请求链接为：https://www.baidu.com
import requests
import json
r = requests.get('https://www.baidu.com/')
print(r.status_code) #状态码
print(r.text) #响应体
print(r.headers) #响应头200
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>???o|??????????°±?￥é</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=???o|???? class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>?°é?</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>?°??</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>è§é￠</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>è′′?§</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>????</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">????</a>');
</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">?′?¤?o§?</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>?3?o???o|</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>????¨???o|???èˉ?</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>?è§?é|</a> ?o?ICPèˉ030173?· <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
{'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Sat, 16 Jul 2022 11:15:27 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:23:55 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}
-
抓取网页数据
import requests
import re
r = requests.get("https://ssr1.scrape.center/")
pattern = re.compile('<h2.*?>(.*?)</h2>',re.S)
titles = re.findall(pattern,r.text)
print(titles)运行结果：
['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption', '泰坦尼克号 - Titanic', '罗马假日 - Roman Holiday', '唐伯虎点秋香 - Flirting Scholar', '乱世佳人 - Gone with the Wind', '喜剧之王 - The King of Comedy', '楚门的世界 - The Truman Show', '狮子王 - The Lion King']
上述利用正则表达式抓取该网站的标题，后面会更新学习正则表达式的相关内容。
-
抓取二进制数据
图片、音频、视频等文件本质上都是由二进制码组成的，有特定的保存格式和对应的解码格式，如果想抓取这些数据，就必须抓取他们的二进制数据。
import requests
r = requests.get("https://ssr1.scrape.center/static/img/logo.png")
print(r.text)
print(r.content)
运行结果部分截图：
?
?对比response的两个属性，注意到r.text出现乱码，而r.content的前面带有一个b，代表是bytes类型的数据，现将提取出的数据进行保存。
import requests
r = requests.get("https://ssr1.scrape.center/static/img/logo.png")
with open('logo.png','wb') as f: #第一个参数的文件名称，第二个参数是以二进制写的形式打开文件
f.write(r.content)2.POST请求
使用requests实现post请求也十分简单，实例如下：
import requests
data = {
"name":"LZQ",
"age" : 22
}
r = requests.post('https://www.httpbin.org/post',params=data)
print(r.text)运行结果：
{
"args": {
"age": "22",
"name": "LZQ"
},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Content-Length": "0",
"Host": "www.httpbin.org",
"User-Agent": "python-requests/2.27.1",
"X-Amzn-Trace-Id": "Root=1-62d29f4b-17b7d6b13b6d5c8d4a65f987"
},
"json": null,
"origin": "120.238.218.103",
"url": "https://www.httpbin.org/post?name=LZQ&age=22"
}
3.响应
请求发送后会得到响应，可以用来获得状态码、响应头、url等其他信息。
import requests
r = requests.get("https://www.baidu.com")
print(type(r.status_code),r.status_code) #状态码
print(type(r.headers),r.headers) #响应头
print(type(r.cookies),r.cookies) #Cookie
print(type(r.url),r.url)
print(type(r.history),r.history) #历史记录运行结果：
<class 'int'> 200
<class 'requests.structures.CaseInsensitiveDict'> {'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Sat, 16 Jul 2022 11:26:22 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:23:55 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}
<class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
<class 'str'> 百度一下，你就知道
<class 'list'> []