[Python知识库]2021-10-10

笔记cocos论坛（爬虫）

爬取标题，访问量，评论数，ID，回复量

import requests
import csv
fp = open("./cocos论坛.csv", 'a',newline='',encoding='utf-8')
writer = csv.writer(fp)
writer.writerow(('id','标题','评论量','点赞量','回复量'))
headers={
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36"
}
page=int(input("请输入需要多少页数据："))
for j in range(1,page+1):
    url=f'https://forum.cocos.org/top.json?ascending=false&order=default&page={j}&per_page=50'
    response=requests.get(url=url,headers=headers).json()
    r=response["topic_list"]['topics']
    for i in r:
        id=i['id']
        title=i['title']
        view=i['views']
        like_count=i['like_count']
        reply_count=i['reply_count']
        print(id,title,view,like_count,reply_count)
        writer.writerow((id,title,view,like_count,reply_count))
fp.close()

结果