|
python
豆瓣电影影评爬取
需要库
lxml，requests
代码如下:
import requests
from lxml import etree
'''定义头部信息'''
head={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"}
base_url='https://movie.douban.com/review/best/?start='
list1=[]
for i in range(5):
url=base_url+'{}'.format(20*i)
r=requests.get(url,headers=head)
domo=r.content.decode('utf8')
jiexi=etree.HTML(domo)
_url=jiexi.xpath('//div[@class="main-bd"]/h2/a/@href')
list1.append(_url)
for _url in list1:
for url1 in _url:
print(url1)
运行结果如下:
https://movie.douban.com/review/13714022/
https://movie.douban.com/review/13715350/
https://movie.douban.com/review/13717055/
https://movie.douban.com/review/13717208/
https://movie.douban.com/review/13717138/
https://movie.douban.com/review/13722389/
https://movie.douban.com/review/13719592/
https://movie.douban.com/review/13721133/
https://movie.douban.com/review/13713602/
https://movie.douban.com/review/13723296/
https://movie.douban.com/review/13721586/
https://movie.douban.com/review/13721525/
https://movie.douban.com/review/13723054/
https://movie.douban.com/review/13722265/
https://movie.douban.com/review/13721537/
https://movie.douban.com/review/13721269/
https://movie.douban.com/review/13720810/
https://movie.douban.com/review/13719412/
https://movie.douban.com/review/13721150/
https://movie.douban.com/review/13723536/
https://movie.douban.com/review/13719679/
https://movie.douban.com/review/13723211/
https://movie.douban.com/review/13717463/
https://movie.douban.com/review/13704260/
https://movie.douban.com/review/13723166/
https://movie.douban.com/review/13720398/
https://movie.douban.com/review/13704362/
https://movie.douban.com/review/13721132/
https://movie.douban.com/review/13719711/
https://movie.douban.com/review/13719578/
https://movie.douban.com/review/13705014/
https://movie.douban.com/review/13717464/
https://movie.douban.com/review/13718000/
https://movie.douban.com/review/13717991/
https://movie.douban.com/review/13717989/
https://movie.douban.com/review/13720534/
https://movie.douban.com/review/13721967/
https://movie.douban.com/review/13722851/
https://movie.douban.com/review/13717445/
https://movie.douban.com/review/13719860/
https://movie.douban.com/review/13716410/
https://movie.douban.com/review/13709577/
https://movie.douban.com/review/13721914/
https://movie.douban.com/review/13719582/
https://movie.douban.com/review/13719572/
https://movie.douban.com/review/13716466/
https://movie.douban.com/review/13713204/
https://movie.douban.com/review/13718758/
https://movie.douban.com/review/13716407/
https://movie.douban.com/review/13713076/
|