[系统运维]Python 解析log日志

Python 解析log日志

• 软件环境
• 环境搭建
• 待解析log日志格式
• log解析脚本
• 解析后文本格式

软件环境

环境搭建

在终端命令行输入以下指令

sudo apt install python3.8 # 安装python3
python3.8 -m pip install python-dateutil # 安装 dateutil 包

待解析log日志格式

log日志一般具有时间戳、事件类型、事件产生对象、属性等，本文以具有下列形式的log日志为例

[2021-09-03 11:03:11]************************** EVENT ERROR **************************
[2021-09-03 11:03:11] DevID:  1
[2021-09-03 11:03:11] Attribute:  187

[2021-09-03 11:05:17]************************** EVENT HEART **************************
[2021-09-03 11:05:17] DevID:  1
[2021-09-03 11:05:17] Attribute:  198

[2021-09-03 11:06:20]************************** EVENT HEART **************************
[2021-09-03 11:06:20] DevID:  3
[2021-09-03 11:06:20] Attribute:  14

[2021-09-03 11:10:41]************************** EVENT HEART **************************
[2021-09-03 11:10:41] DevID:  5
[2021-09-03 11:10:41] Attribute:  96

[2021-09-03 11:16:34]************************** EVENT HEART **************************
[2021-09-03 11:16:34] DevID:  1
[2021-09-03 11:16:34] Attribute:  153

log解析脚本

import re
from dateutil.parser import *

DEVID = 1
TAB = "    "

def logparser():
    line_num = 0
	
	# locate EVENT HEART
    pattern_heart = re.compile(r'EVENT HEART')
    # locate xxxx-xx-xx xx:xx:xx
    pattern_timestamp=re.compile(r'[0-9]*-[0-9]*-[0-9]* [0-9]*:[0-9]*:[0-9]*')

    last_time = parse("2021-09-03 00:00:00")
    diff_time = parse("2021-09-03 00:00:00")

    outfile = open("./out.txt", "w+")

    with open("./log.txt") as f:
        lines = f.readlines()
        for line in lines:
            # look for event heart
            result_heart = pattern_heart.findall(line)

            # no heart event in this line
            if not result_heart:   
                line_num = line_num + 1
                continue;

            # 2 is the offset line of Attribute
            if line_num + 2 > len(lines):   
                print("END")
                return;

            # look for DevID, 1 is the offset line of DevID
            DevID = lines[line_num + 1].split(":")[-1].strip()

            # check DevID
            if DevID != str(DEVID):
                line_num = line_num + 1
                continue;

            # 2 is the offset line of Attribute
            Attribute = lines[line_num + 2].split(":")[-1].strip()
            
            # look for string according to regex pattern
            timestamps = pattern_timestamp.findall(line)
            # exist some matches
            if timestamps:
                for timestamp in timestamps:
                    timestamp = parse(timestamp)
                    diff_time = timestamp - last_time
                    last_time = timestamp
                    outfile.write(str(timestamp)+TAB+str(diff_time)+TAB+Attribute+"\n")
                line_num = line_num + 1
            else:
                line_num = line_num + 1
                print("FORMAT ERROR")
                
        f.close()
    outfile.close()

if __name__ == "__main__":
    logparser()
    print("DONE")

解析后文本格式

2021-09-03 11:05:17    11:05:17    198
2021-09-03 11:16:34    0:11:17    153