一.wireshark抓取网络数据包(疯狂聊天室)
1.下载并打开疯狂聊天室程序
![在这里插入图片描述](https://img-blog.csdnimg.cn/d926f23f18184a68b2eab21c3aa6f8ae.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16) 两台设备输入相同的房间号并发送信息 ![在这里插入图片描述](https://img-blog.csdnimg.cn/b4cd806e94f54a6ab00ca51d1d0ddbe2.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/93fd1515003c41338ea49a0b8c30f009.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
2.wireshark抓包
![在这里插入图片描述](https://img-blog.csdnimg.cn/27c722d0e4e2446e804ad8e56bd32024.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16) ![在这里插入图片描述](https://img-blog.csdnimg.cn/1d197694cbf44f9894e1da687523227e.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/e386c4ca935047d8a5b81815d340545f.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16) 对比三条信息可以看到:
- 此程序网络连接采用的是UDP协议和6234端口号
- 发送英文和数字信息可以看到抓取的信息,但是中文信息是乱码,说明汉字可能经过了某种编码转换
二.对南阳理工学院ACM题目网站练习题目数据的抓取和保存
1.打开目标网站,查看代码
![在这里插入图片描述](https://img-blog.csdnimg.cn/597c166ca601409e9c7a40a44f3b8207.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
2.利用jupyter爬取
import requests
from bs4 import BeautifulSoup
import csv
from tqdm import tqdm
Headers = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3741.400 QQBrowser/10.5.3863.400'
csvHeaders = ['题号', '难度', '标题', '通过率', '通过数/总提交数']
subjects = []
print('题目信息爬取中:\n')
for pages in tqdm(range(1, 1 + 1)):
r = requests.get(f'http://www.51mxd.cn/problemset.php-page={pages}.htm', Headers)
r.raise_for_status()
r.encoding = 'utf-8'
soup = BeautifulSoup(r.text, 'html5lib')
td = soup.find_all('td')
subject = []
for t in td:
if t.string is not None:
subject.append(t.string)
if len(subject) == 5:
subjects.append(subject)
subject = []
with open('NYOJ_Subjects.csv', 'w', newline='') as file:
fileWriter = csv.writer(file)
fileWriter.writerow(csvHeaders)
fileWriter.writerows(subjects)
print('\n题目信息爬取完成!!!')
运行结果: ![在这里插入图片描述](https://img-blog.csdnimg.cn/1aeb65e147d645b3ac59882abf3a5943.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16) ![在这里插入图片描述](https://img-blog.csdnimg.cn/5a6d5b5d79644b1fb664f288d389f720.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
三.爬取重庆交通大学新闻网站中近几年所有的信息通知的发布日期和标题全部爬取下来,并写到CSV电子表格中
1.打开网站
![在这里插入图片描述](https://img-blog.csdnimg.cn/6f829ceca0944ca5b9858e570e4322e1.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
2.运行代码爬取
import requests
from bs4 import BeautifulSoup
import csv
from tqdm import tqdm
import urllib.request, urllib.error
subjects = []
Headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 Edg/95.0.1020.53"
}
csvHeaders = ['时间', '标题']
print('信息爬取中:\n')
for pages in tqdm(range(1, 65 + 1)):
request = urllib.request.Request(f'http://news.cqjtu.edu.cn/xxtz/{pages}.htm', headers=Headers)
html = ""
try:
response = urllib.request.urlopen(request)
html = response.read().decode("utf-8")
except urllib.error.URLError as e:
if hasattr(e, "code"):
print(e.code)
if hasattr(e, "reason"):
print(e.reason)
soup = BeautifulSoup(html, 'html5lib')
subject = []
li = soup.find_all('li')
for l in li:
if l.find_all('div',class_="time") is not None and l.find_all('div',class_="right-title") is not None:
for time in l.find_all('div',class_="time"):
subject.append(time.string)
for title in l.find_all('div',class_="right-title"):
for t in title.find_all('a',target="_blank"):
subject.append(t.string)
if subject:
print(subject)
subjects.append(subject)
subject = []
with open('test.csv', 'w', newline='') as file:
fileWriter = csv.writer(file)
fileWriter.writerow(csvHeaders)
fileWriter.writerows(subjects)
print('\n信息爬取完成!!!')
![在这里插入图片描述](https://img-blog.csdnimg.cn/7cc1f6f5375e4f42a4a7242932c71af0.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16) ![在这里插入图片描述](https://img-blog.csdnimg.cn/b42bed323b6c432ea145efe197a9a515.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBAPy9Ta3k_Pz8_,size_20,color_FFFFFF,t_70,g_se,x_16)
总结
爬虫从初始网页的URL开始,获取初始网页上的URL,在抓取网页的过程中,不断从当前页面上抽取新的URL放X队列,直到满足系统给定的停止条件。
参考
https://blog.csdn.net/qq_47281915/article/details/121344160 https://blog.csdn.net/m0_51120713/article/details/121379942
|