Gevent是一个基于微线程库Greenlet的并发框架
优点 只需要monkey.patch_all(),就能自动修改阻塞为非阻塞
缺点 不知道具体patch了哪些库修改了哪些模块、类、函数,出问题不好调试
安装
pip install gevent
demo
import gevent
from bs4 import BeautifulSoup
from gevent import monkey
monkey.patch_all()
import requests
def get_page_source(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
title_list = soup.find('div', attrs={'id': 'post_list'}).find_all('a', attrs={'class': 'post-item-title'})
for title in title_list:
print(title['href'], title.text)
if __name__ == '__main__':
threads = []
for i in range(1, 10):
page = f'https://www.cnblogs.com/#p{i}'
threads.append(gevent.spawn(get_page_source, page))
gevent.joinall(threads)
还一点说明，在导入的时候，先导入gevent，然后猴子打补丁，然后在requests 不然会报错，ssl和递归的错误
