今天我们想实现豆瓣电影排行爬取 如上图所示,我们希望将电影的相关信息通过爬虫爬取下来,并写入文档。
实现过程
import requests
from lxml import etree
以上需要安装requests和lxml两个库,可以通过库安装完成
headers={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'
}
j=1
fl=open("电影排行.doc","w",encoding='utf-8')
上述实现HTTP请求头headers和文档的建立
for i in range(0,250,25):
url='https://movie.douban.com/top250?start='+str(i)+'&filter='
response=requests.get(url,headers=headers)
text=response.text
html=etree.HTML(text.encode('utf-8'))
ul=html.xpath('//div[@class="info"]')
for div in ul:
t="".join(div.xpath('./div[@class="hd"]/a/span[@class="title"]/text()'))
s="".join(div.xpath('./div[@class="hd"]/a/span[@class="other"]/text()'))
title=t+s
d="".join(div.xpath('./div[@class="bd"]/p/text()'))
director=d.replace(" ","").replace("\n","").replace("主演","\n主演")
quote="".join(div.xpath('./div[@class="bd"]/p[@class="quote"]/span[@class="inq"]/text()'))
content=str(j)+"."+title+"\n"+director+"\n主题:"+quote+"\n"
fl.write(content)
j=j+1
fl.close()
上述代码就实现了排行的爬取。 其中:
url='https://movie.douban.com/top250?start='+str(i)+'&filter='
可以通过翻页可以发现,此排行榜每一页的的url格式都是以上方式。
response=requests.get(url,headers=headers)
text=response.text
html=etree.HTML(text.encode('utf-8'))
ul=html.xpath('//div[@class="info"]')
for div in ul:
t="".join(div.xpath('./div[@class="hd"]/a/span[@class="title"]/text()'))
s="".join(div.xpath('./div[@class="hd"]/a/span[@class="other"]/text()'))
title=t+s
d="".join(div.xpath('./div[@class="bd"]/p/text()'))
director=d.replace(" ","").replace("\n","").replace("主演","\n主演")
quote="".join(div.xpath('./div[@class="bd"]/p[@class="quote"]/span[@class="inq"]/text()'))
以上代码部分都是实现对文本内容进行定位,这个需要通过右击鼠标,选择检查,查看html代码找到。 在此部分中使用.join()函数是将文本内容进行连接。 具体.join()函数的使用方法如下:
str = "-";
seq = ("a", "b", "c");
print (str.join( seq ));
在此部分中使用.replace()函数是将文本内容进行替换。 具体.replace()函数的使用方法如下:
str = "this is string example....wow!!! this is really string";
print(str.replace("is", "was"));
print(str.replace("is", "was", 3));
完整代码
import requests
from lxml import etree
headers={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'
}
j=1
fl=open("电影排行.doc","w",encoding='utf-8')
for i in range(0,250,25):
url='https://movie.douban.com/top250?start='+str(i)+'&filter='
response=requests.get(url,headers=headers)
text=response.text
html=etree.HTML(text.encode('utf-8'))
ul=html.xpath('//div[@class="info"]')
for div in ul:
t="".join(div.xpath('./div[@class="hd"]/a/span[@class="title"]/text()'))
s="".join(div.xpath('./div[@class="hd"]/a/span[@class="other"]/text()'))
title=t+s
d="".join(div.xpath('./div[@class="bd"]/p/text()'))
director=d.replace(" ","").replace("\n","").replace("主演","\n主演")
quote="".join(div.xpath('./div[@class="bd"]/p[@class="quote"]/span[@class="inq"]/text()'))
content=str(j)+"."+title+"\n"+director+"\n主题:"+quote+"\n"
fl.write(content)
j=j+1
fl.close()
实现效果:
|