[Python知识库] 案例逐步演示python利用正则表达式提取指定内容并输出到csv

开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> Python知识库 -> 案例逐步演示python利用正则表达式提取指定内容并输出到csv -> 正文阅读

[Python知识库]案例逐步演示python利用正则表达式提取指定内容并输出到csv

背景和目标

这次我想要处理的是一个txt文件，里面的内容是一台机器定时ping另一台机器的输出结果，想要提取出的内容是时间和rtt值，最后还要把结果输出到csv文件。

1. 明确要提取的内容，编写正则表达式

要提取的文本如下：
在这里插入图片描述
第一步是要编写正则表达式，此时可以先不要读取数据文件。先复制一部分数据到str中，方便测试。
编写正则表达式用到了re模块，因为每个人要处理的文本是不一样的，所以需要自己去学习基本的使用方法。re具体使用方法可以参考这篇文章：
https://zhuanlan.zhihu.com/p/139596371

关键就是弄清楚.*?和{}的作用，还有re.S可以匹配到换行符，就可以比较容易地写出正确的表达式。

import re
# 为了方便测试，我把一部分文本先放到str里
str='''
2022-03-11 15:21:48
1
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=253 ms

--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9000ms
rtt min/avg/max/mdev = 250.203/250.563/253.202/0.961 ms
2022-03-11 15:22:40
2
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=250 ms

--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9009ms
rtt min/avg/max/mdev = 250.181/250.256/250.434/0.636 ms
2022-03-11 15:23:44
3
PING 81.71.51.181 (81.71.51.181) 56(84) bytes of data.
64 bytes from 81.71.51.181: icmp_seq=1 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=2 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=3 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=4 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=5 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=6 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=7 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=8 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=9 ttl=45 time=250 ms
64 bytes from 81.71.51.181: icmp_seq=10 ttl=45 time=250 ms

--- 81.71.51.181 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9009ms
rtt min/avg/max/mdev = 250.209/250.320/250.658/0.563 ms
'''

# print(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2})', str))  # 提取时间
# print(re.findall(r'mdev = (.*?) ms', str))  # 提取rtt

print(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', data, re.S))  # 提取时间和rtt 包括换行

输出：

D:\python37\python.exe D:/test/data_process.py
['2022-03-11 15:21', '2022-03-11 15:22', '2022-03-11 15:23']
['250.203/250.563/253.202/0.961', '250.181/250.256/250.434/0.636', '250.209/250.320/250.658/0.563']
[('2022-03-11 15:21', '250.203/250.563/253.202/0.961'), ('2022-03-11 15:22', '250.181/250.256/250.434/0.636'), ('2022-03-11 15:23', '250.209/250.320/250.658/0.563')]

Process finished with exit code 0

2. 从文件中读入数据

编写出正确的正则表达式后，就可以从文件中读数据了

import re
# 读取文件
with open("ping/ping_flkf_gz.txt","r") as input_file:
    str = input_file.read()

print(re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', str, re.S))  # 提取时间和延迟 包括换行

input_file.close()  # 关闭文件

输出比较多，截取一部分展示：

D:\python37\python.exe D:/test/data_process.py
[('2022-03-11 15:21', '250.203/250.563/253.202/0.961'), ('2022-03-11 15:22', '250.181/250.256/250.434/0.636'), ('2022-03-11 15:23', '250.209/250.320/250.658/0.563'), ('2022-03-11 15:25', '250.183/250.240/250.275/0.225'), ('2022-03-11 15:26', '250.217/250.240/250.300/0.592'), ('2022-03-11 15:27', '250.166/250.362/250.956/0.683'), ('2022-03-11 15:28', '250.186/250.256/250.343/0.319'), ('2022-03-11 15:29', '250.181/250.435/252.077/0.776'), ('2022-03-11 15:30', '250.177/250.249/250.401/0.673'), ('2022-03-11 15:31', '250.210/250.436/251.498/0.376'), ('2022-03-11 15:32', '250.207/250.280/250.588/0.401'), ('2022-03-11 15:33', '250.237/250.336/250.747/0.568'), ('2022-03-11 15:34', '250.217/250.283/250.437/0.675'), ('2022-03-11 15:35', '250.254/250.456/251.092/0.623'), ('2022-03-11 15:36', '250.167/250.236/250.308/0.226'), ('2022-03-11 15:37', '250.162/250.399/251.032/0.667'), ('2022-03-11 15:38', '250.207/250.261/250.406/0.053'), ('2022-03-11 15:39', '250.219/250.657/252.056/0.878')]

这里其实是一个列表，里面的每个元组是我提取出来的时间和rtt。

3. 写入csv

能够正确读取输入文件并提取数据后，下一步就是要把结果写入csv文件，所以用到了csv模块。
for循环遍历列表，使用csv_writer.writerow一行行写入csv文件。

import re
import csv

# 读取文件
with open("ping/ping_flkf_gz.txt", "r") as input_file:
    str = input_file.read()

# 用一个列表接收提取出来的数据
list = re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?) ms', str, re.S)  

# 1.创建文件对象
output_file = open('res/ping/ping_flkf_gz.csv', 'w', encoding='utf-8', newline='')
# 2.基于文件对象构建csv写入对象
csv_writer = csv.writer(output_file)
# 3. 写入表头
csv_writer.writerow(["time", "latency"])
# 4.遍历列表，写入csv文件
for i in list:
    csv_writer.writerow([i[0], i[1]])

input_file.close()  # 关闭文件
output_file.close()  # 关闭文件

结果就写入到csv文件中了

time,latency
2022-03-11 15:21,250.203/250.563/253.202/0.961
2022-03-11 15:22,250.181/250.256/250.434/0.636
2022-03-11 15:23,250.209/250.320/250.658/0.563
2022-03-11 15:25,250.183/250.240/250.275/0.225
2022-03-11 15:26,250.217/250.240/250.300/0.592
2022-03-11 15:27,250.166/250.362/250.956/0.683
2022-03-11 15:28,250.186/250.256/250.343/0.319
2022-03-11 15:29,250.181/250.435/252.077/0.776
2022-03-11 15:30,250.177/250.249/250.401/0.673
2022-03-11 15:31,250.210/250.436/251.498/0.376

4. 还可以把每个数值分开存放

发现此时latency那一列是这样的250.203/250.563/253.202/0.961
为了后面方便处理，把每个数值单独作为一列，因此要修改正则表达式

import re
import csv

# 读取文件
with open("ping/ping_flkf_gz.txt", "r") as input_file:
    str = input_file.read()

# 用一个列表接收提取出来的数据 这里修改了正则表达式，使得每个数值单独作为一列
list = re.findall(r'(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}).*?mdev = (.*?)/(.*?)/(.*?)/(.*?) ms', str, re.S)

# 1.创建文件对象
output_file = open('res/ping/ping_flkf_gz.csv', 'w', encoding='utf-8', newline='')
# 2.基于文件对象构建csv写入对象
csv_writer = csv.writer(output_file)
# 3. 写入表头
csv_writer.writerow(["time", "min", "avg", "max", "mdev"])
# 4.遍历列表，写入csv文件
for i in list:
    csv_writer.writerow([i[0], i[1], i[2], i[3], i[4]])

input_file.close()  # 关闭文件
output_file.close()  # 关闭文件