[Python知识库]python利用bs4获取网络小说

所选定的小说网址：

https://www.shicimingju.com/book/xiyouji.html

如果想更换其他小说资源：

在上一级网址就行：

https://www.shicimingju.com/book

代码如下：

from bs4 import BeautifulSoup
import requests
if __name__=="__main__":
    url='https://www.shicimingju.com/book/xiyouji.html'  
    headers={'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1'}
    page_text=requests.get(url=url,headers=headers)
    (page_text).encoding='gzip'
    page_text=(page_text).text
    soup=BeautifulSoup(page_text,'lxml')
    li_list=soup.select('.book-mulu > ul > li')

    fp=open('./西游记.txt','w',encoding='utf-8')
    for li in li_list:
        title=li.a.string
        detail_url='https://www.shicimingju.com'+li.a['href']
        detail_page_text=requests.get(url=detail_url,headers=headers)
        (detail_page_text).encoding='gzip'
        detail_page_text=(detail_page_text).text
        detail_soup=BeautifulSoup(detail_page_text,'lxml')
        div_tag=detail_soup.find('div',class_='chapter_content')
        content=div_tag.text
        fp.write(title+':'+content+'\n')
        print(title,"下载成功！")
    fp.close()

下载结果输出：

最后存储于txt文本中：