[Python知识库]python爬虫获取豆瓣TOP250电影的详细信息

# encoding:utf-8
# encoding:unicode_escape
import time
import codecs
import requests
import lxml.html

with codecs.open("movies.txt", "w", "utf-8") as f:
    myheaders = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"}     
    url_tpl = "https://movie.douban.com/top250?start={}&filter="

    for page in range(10):
        # 打印一些信息显示抓取进度
        print("get page {}".format(page+1))
        start = page * 25
        url = url_tpl.format(start)

        # 替换抓取的网址为生成的URL
        http_response = requests.get(url, headers=myheaders)
        http_response.encoding = "utf-8"
        html = lxml.html.fromstring(http_response.text)
        movies = html.xpath('//*[@id="content"]/div/div[1]/ol/li')

        for movie in movies:
            # 强制转换为python字符串
            movie_text = str(movie.text_content())
            # 去掉换行符，保证数据文本都输出在一行
            clean_movie_text = movie_text.replace("\n", "")
            # 直接将文本打印到打开的文件里
            print(clean_movie_text, file=f)
            
        # 根据豆瓣电影的robots.txt，暂停5秒
        time.sleep(5)

运行过程

由于豆瓣TOP250的电影网页每一页只展示25个电影，因此需要爬取共10页。

完成后会在py代码的共同目录下生成一个movies.txt文档。
文档里面就是TOP250的详细信息了：