一、systemd-journald服务

1.systemd-journald介绍

# 系统时间日志的核心服务
# 收集包括来自内核，早期引导阶段啊，进程启动，运行时的标准输出和错误输出，以及syslog时间等
# 保存在二进制的日志文件中，具有易失性，重启后不保存

2.系统日志默认保存路径

保存目录为/run/log/journal

[root@tianyi 4f596c775d924b618367d1c448fd5578]# pwd
/run/log/journal/4f596c775d924b618367d1c448fd5578
[root@tianyi 4f596c775d924b618367d1c448fd5578]# ls
system@f51ef91c90ff4fd48133fe727841edb8-000000000011b8bf-0005c7be331e995d.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000011d6a0-0005c7c4724c57e6.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000011f42f-0005c7cab33cc55b.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000121201-0005c7cf0099fa95.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000122fca-0005c7db1bc72be2.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000124d63-0005c7e173a7688d.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000126bf3-0005c7e70d3d55d0.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000128a4c-0005c7ebf2c2f08f.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012a845-0005c7ef788f2d0b.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012c602-0005c7f31a53c98f.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012e3d8-0005c7f8cdf9eace.journal
system.journal

3.systemd-journald的配置文件修改

①配置文件目录

/etc/systemd/journald.conf

②配置文件修改

persistent: 将日志存储在/var/log/journal目录中，若该目录不存在，则systemd-journald服务自动创建。
volatile：将日志存储在易失性的目录/run/log/journal目录中，若该目录不存在，则systemd-journald服务自动创建。
auto：如果/var/log/journal目录存在，那么rsyslog会使用持久存储，否则为易失性存储，此为默认配置。

[root@tianyi systemd]# grep auto journald.conf 
#Storage=auto

4.检索日志消息

①journalctl命令的相关选项

-n 指定显示末尾几条消息，默认最后10条日志消息
-f 与tail -f 命令相似
-p 指定显示某个优先级以上的日志
--since或者--until   限制特定的时间段，时间格式为“YYYY-MM-DD hh:mm:ss”等
-o verbose 查看日志的详细信息
_PID 进程的PID
_UID  运行该进程的用户ID
_SYSTEMD_UNIT  启动该进程的systemd单元
_COMM 指定命令的名称
_EXE 京城的可执行文件的路径

②示例

[root@tianyi systemd]# journalctl -n 5  _SYSTEMD_UNIT=sshd.service
-- Logs begin at Fri 2021-07-23 06:56:38 CST, end at Mon 2021-07-26 23:20:45 CST. --
Jul 26 23:20:33 tianyi sshd[10011]: pam_unix(sshd:auth): check pass; user unknown
Jul 26 23:20:33 tianyi sshd[10011]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.117.46.219
Jul 26 23:20:36 tianyi sshd[10011]: Failed password for invalid user vbox from 1.117.46.219 port 41028 ssh2
Jul 26 23:20:36 tianyi sshd[10011]: Received disconnect from 1.117.46.219 port 41028:11: Bye Bye [preauth]
Jul 26 23:20:36 tianyi sshd[10011]: Disconnected from invalid user vbox 1.117.46.219 port 41028 [preauth

二、rsyslog服务

1.日志保存相关路径

/var/log/message 系统绝大多数的日志文件都会记录在该文件中
/var/log/secure 记录与安全相关的日志，比如创建用户，修改密码，切换用户等操作
/var/log/cron 记录与计划任务相关
/var/log/maillog 记录与邮件相关的日志
/var/log/boot.log 记录与启动相关的

2.日志消息的类型

lpr: 打印相关的日志
auth：认证相关的日志
user：用户相关的日志
cron：计划任务相关的日志
kern：内核相关的日志
mail：邮件相关的日志
daemon:系统服务相关的日志
authpri：授权相关的日志
security：安全相关的日志
local0-local7:自定义相关的日志信息

3.日志消息优先级


优先级	代码	严重性
none	无	不记录任何信息
emerg	0	内核崩溃等严重信息
alert	1	需要立即修改的信息
crit	2	严重错误级别
err	3	错误级别
warning	4	警告级别
notice	5	具有重要性的普通信息
info	6	一般信息
warning	7	调试信息

4.自定义日志规则

①日志定义的格式

facility.priority Target

mail.info /var/log/maillog 比指定级别更高的日志级别，包括级别自身，保存到 /var/log/maillog
mail.=info /var/log/maillog 明确指定级别为info,保存到/var/log/maillog 
mail.！info /var/log/maillog 除了指定的日志级别以外的级别，都保存到/var/log/maillog 
*.info info /var/log/maillog 所有类型的info级别都保存到/var/log/maillog
mail.*  /var/log/maillog mail类型的所有级别日志都保存到/var/log/maillog

三、日志查看

[root@tianyi log]# grep -E -C 5 '(err|not|no)' messages
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start timed out.
Jul 25 05:44:39 tianyi systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device.
Jul 25 05:44:39 tianyi systemd[1]: Dependency failed for /dev/disk/by-uuid/c30fcda5-d830-4c90-b818-831e33389b2e.
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.swap: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.swap/start failed with result 'dependency'.
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start failed with result 'timeout'.
Jul 25 07:40:57 tianyi rsyslogd[1750]: imjournal: sd_journal_get_cursor() failed: Cannot assign requested address [v8.37.0-9.el8]
Jul 25 07:40:57 tianyi rsyslogd[1750]: imjournal: journal reloaded... [v8.37.0-9.el8 try http://www.rsyslog.com/e/0 ]
Jul 25 07:41:56 tianyi systemd[1]: Starting dnf makecache...
Jul 25 07:42:02 tianyi dnf[6483]: Docker CE Stable - x86_64                       690  B/s | 3.5 kB     00:05
Jul 25 07:42:07 tianyi dnf[6483]: Zabbix                                          590  B/s | 2.9 kB     00:05
Jul 25 07:42:12 tianyi dnf[6483]: huawei-AppStream                                875  B/s | 4.3 kB     00:05
--
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info>  [1627190262.1609] dhcp4 (eth0):   hostname 'host-192-168-1-209'
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info>  [1627190262.1609] dhcp4 (eth0):   gateway 192.168.1.1
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info>  [1627190262.1609] dhcp4 (eth0):   static route 169.254.169.254/32 gw 192.168.1.254
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info>  [1627190262.1609] dhcp4 (eth0):   mtu 1500
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info>  [1627190262.1611] dhcp4 (eth0): state changed bound -> bound
Jul 25 13:17:42 tianyi dbus-daemon[614]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.33' (uid=0 pid=1936 comm="/usr/sbin/NetworkManager --no-daemon ")
Jul 25 13:17:42 tianyi systemd[1]: Starting Network Manager Script Dispatcher Service...
Jul 25 13:17:42 tianyi dbus-daemon[614]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jul 25 13:17:42 tianyi systemd[1]: Started Network Manager Script Dispatcher Service.
Jul 25 13:17:42 tianyi nm-dispatcher[11673]: req:1 'dhcp4-change' [eth0]: new request (5 scripts)
Jul 25 13:17:42 tianyi nm-dispatcher[11673]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
Jul 25 13:19:12 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start timed out.
Jul 25 13:19:12 tianyi systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device.

四、日志服务器搭建

1.日志服务器介绍

rstslog是一个C/S架构，可以基于TCP和UDP工作，默认监听端口为514

2.服务端配置

①修改服务端配置文件

[root@IT-01 log]# vim  /etc/rsyslog.conf 
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

②重启服务及防火墙关闭

[root@IT-01 log]# systemctl restart rsyslog.service 
[root@IT-01 log]# systemctl stop firewalld

3.客户端配置

①修改配置文件

 vim  /etc/rsyslog.conf 

*.info                        @192.168.200.129

②重启服务

[root@node1 ~]# systemctl restart rsyslog.service

4.测试日志

服务端查看

[root@IT-01 log]# tail -n 5 /var/log/messages
Jul 27 00:03:02 node1 dnf[3099]: Repository AppStream is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository BaseOS is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository AppStream is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository BaseOS is listed more than once in the configuration
Jul 27 00:03:13 node1 dnf[3099]: CentOS-8 - AppStream - mirrors.aliyun.com       0.0  B/s |   0  B     00:10

五、日志轮询

1.logrotate介绍

logrotate工具会轮询日志文件，防止日志文件占用过多的系统空间。

配置文件
[root@IT-01 log]# /etc/logrotate.d/*
[root@IT-01 log]# /etc/logrotate.conf

2.日志目录查看

[root@IT-01 log]# ls
anaconda           cron                      firewalld            maillog-20210623   qemu-ga            spooler-20210725      vmware-network.9.log
audit              cron-20210623             gdm                  maillog-20210707   rhsm               sssd                  vmware-network.log
boot.log           cron-20210707             glusterfs            maillog-20210717   samba              swtpm                 vmware-vgauthsvc.log.0
boot.log-20210506  cron-20210717             hawkey.log           maillog-20210725   secure             tuned                 vmware-vmsvc-root.log
boot.log-20210507  cron-20210725             hawkey.log-20210623  messages           secure-20210623    vmware                vmware-vmtoolsd-root.log
boot.log-20210508  cups                      hawkey.log-20210707  messages-20210623  secure-20210707    vmware-network.1.log  vmware-vmusr-root.log
boot.log-20210511  dnf.librepo.log           hawkey.log-20210717  messages-20210707  secure-20210717    vmware-network.2.log  wtmp
boot.log-20210531  dnf.librepo.log-20210623  hawkey.log-20210725  messages-20210717  secure-20210725    vmware-network.3.log  zabbix
boot.log-20210613  dnf.librepo.log-20210707  httpd                messages-20210725  speech-dispatcher  vmware-network.4.log
boot.log-20210722  dnf.librepo.log-20210717  lastlog              mysql              spooler            vmware-network.5.log
btmp               dnf.librepo.log-20210725  libvirt              php-fpm            spooler-20210623   vmware-network.6.log
btmp-20210707      dnf.log                   mail                 ppp                spooler-20210707   vmware-network.7.log
chrony             dnf.rpm.log               maillog              private            spooler-20210717   vmware-network.8.log