[Python知识库]Python利用get、post请求爬取数据并写入Excel表格

要求：人民邮电出版社（http://www.ptpress.com.cn）主页

1、获取“新书推荐模块”中设计类相关图书的基本信息（‘bookName’，‘bookId’，‘picPath’）保存到Excel中，
2、并且在上一步的基础上爬取图书的价格和库存信息，保存到Excel中。

一、python将数据写入Excel表格

将数据写入Excel表格利用的是openpyxl，直接pip install openpyxl即可安装

#创建工作簿
book = Workbook()
#激活worksheet
mb = book.active
#利用append写入数据
mb.append([1,2,3])
#保存
book.save('test.xlsx')

二、json.loads() :将json格式数据转换为字典

三、 #coding=gbk ：py文件本身不支持中文，需要把文件编码类型改为UTF-8类型

完整代码如下：

#coding=gbk
# 通过get请求，请求到设计类图书信息
import requests
import json
from openpyxl import Workbook
url = "https://www.ptpress.com.cn/recommendBook/getRecommendBookListForPortal?bookTagId=eaecc2a4-2d23-4193-935a-ab3db4910a5c"
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
    'Accept-Encoding':'gzip, deflate, br',
    'Cookie':'gr_user_id=546172fa-e41a-435a-ae8b-03aa14a54b8b; acw_tc=2760821c16369652778958602e88299a0b9105bad593d33d8f74a59d6aafd2; JSESSIONID=2F2C2EAC9086E016E94F07278A3C9CC1; gr_session_id_9311c428042bb76e=984df283-4b7b-4f25-a9d5-b9d5fb0ab693; gr_session_id_9311c428042bb76e_984df283-4b7b-4f25-a9d5-b9d5fb0ab693=true'
}
# 发送get请求，请求到的数据字典
response = requests.get(url,headers=headers)
tans_dict = json.loads(response.text)
results = tans_dict['data']
print(results)

# ----------------1、设计新书基本信息表---------------
mybook = Workbook()
book = mybook.active
book.append(['书名','编号','图片地址'])
# ----------------2、设计类新书价格库存信息表----------------------

newbook = Workbook()
newbooks=newbook.active
newbooks.append(["折后价格",'库存'])

# 发送post请求，获取图书价格
priceurl='https://www.ptpress.com.cn/bookinfo/getBookDetailsById'
priceheaders={
    'Cookie':'gr_user_id=546172fa-e41a-435a-ae8b-03aa14a54b8b; acw_tc=2760821c16369652778958602e88299a0b9105bad593d33d8f74a59d6aafd2; JSESSIONID=2F2C2EAC9086E016E94F07278A3C9CC1; gr_session_id_9311c428042bb76e=984df283-4b7b-4f25-a9d5-b9d5fb0ab693; gr_session_id_9311c428042bb76e_984df283-4b7b-4f25-a9d5-b9d5fb0ab693=true',
    'Accept-Encoding':'gzip, deflate, br',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
}

# 发送post请求，获取图书库存
stockurl='https://www.ptpress.com.cn/order/getBookSaleStock'
stockheaders={
    'Cookie':'gr_user_id=546172fa-e41a-435a-ae8b-03aa14a54b8b; acw_tc=2760821c16369652778958602e88299a0b9105bad593d33d8f74a59d6aafd2; JSESSIONID=2F2C2EAC9086E016E94F07278A3C9CC1; gr_session_id_9311c428042bb76e=984df283-4b7b-4f25-a9d5-b9d5fb0ab693; gr_session_id_9311c428042bb76e_984df283-4b7b-4f25-a9d5-b9d5fb0ab693=true',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
    'Accept-Encoding':'gzip, deflate, br'
}

for result in results:
    # 将新书信息写入Excel表格
    book.append([result['bookName'],result['bookId'],result['picPath']])

    # ---------------post请求获取图书价格---------------------
    price_form = {
        'bookId': result['bookId']
    }
    priceresponse = requests.post(priceurl, headers=priceheaders, data=price_form)
    priceresponse.encoding = "utf-8"
    price_dict = json.loads(priceresponse.text)
    print(price_dict)

    # ---------------post请求获取图书库存---------------------
    stock_form={
        'isbn':price_dict['data']['bookDetail']['data']['isbn']
    }
    stockresponse = requests.post(stockurl, headers=stockheaders, data=stock_form)
    stockresponse.encoding = "utf-8"
    stock_dict = json.loads(stockresponse.text)
    print(stock_dict)

    newbooks.append([price_dict['data']['discountPrice'],stock_dict['data']])
mybook.save('设计新书基本信息.xlsx')
newbook.save('设计类新书价格库存信息表.xlsx')

结果
设计新书基本信息.xlsx

设计类新书价格库存信息表.xlsx