[大数据]在 CentOS 上部署 Fluent Bit 发送日志到 Kafka

参考我翻译的官方安装文档：Fluent Bit 安装在 Linux

下面部分命令前面添加了sudo，因为使用的不是root账户，很多地方没有权限。

1. 文件上传

先将 td-agent-bit.repo 文件上传到 /etc/yum.repos.d 目录下，文件内容为：

[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1

2. 开始安装

然后执行 yum 安装，中间根据提示需要按两次 y 进行确认：

[test@node-13 local]$ sudo yum install td-agent-bit
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.sjtu.edu.cn
 * extras: ftp.sjtu.edu.cn
 * updates: mirror.lzu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package td-agent-bit.x86_64 0:1.8.2-1 will be installed
--> Processing Dependency: libpq.so.5()(64bit) for package: td-agent-bit-1.8.2-1.x86_64
--> Running transaction check
---> Package postgresql-libs.x86_64 0:9.2.24-7.el7_9 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================================================================================================
 Package                                                      Arch                                                Version                                                        Repository                                                 Size
=================================================================================================================================================================================================================================================
Installing:
 td-agent-bit                                                 x86_64                                              1.8.2-1                                                        td-agent-bit                                              7.3 M
Installing for dependencies:
 postgresql-libs                                              x86_64                                              9.2.24-7.el7_9                                                 updates                                                   235 k

Transaction Summary
=================================================================================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 7.5 M
Installed size: 30 M
Is this ok [y/d/N]: y
Downloading packages:
postgresql-libs-9.2.24-7.el7_9 FAILED                                          
http://mirror.lzu.edu.cn/centos/7.9.2009/updates/x86_64/Packages/postgresql-libs-9.2.24-7.el7_9.x86_64.rpm: [Errno 14] curl#18 - "transfer closed with 214735 bytes remaining to read"                         ]  0.0 B/s |    0 B  --:--:-- ETA 
Trying other mirror.
(1/2): postgresql-libs-9.2.24-7.el7_9.x86_64.rpm                                                                                                                                                                          | 235 kB  00:00:00     
warning: /var/cache/yum/x86_64/7/td-agent-bit/packages/td-agent-bit-1.8.2-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 6ea0722a: NOKEY============================-                                    ] 905 kB/s | 4.7 MB  00:00:03 ETA 
Public key for td-agent-bit-1.8.2-1.x86_64.rpm is not installed
(2/2): td-agent-bit-1.8.2-1.x86_64.rpm                                                                                                                                                                                    | 7.3 MB  00:00:03     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                            2.2 MB/s | 7.5 MB  00:00:03     
Retrieving key from https://packages.fluentbit.io/fluentbit.key
Importing GPG key 0x6EA0722A:
 Userid     : "Eduardo Silva <eduardo@treasure-data.com>"
 Fingerprint: f209 d876 2a60 cd49 e680 633b 4ff8 368b 6ea0 722a
 From       : https://packages.fluentbit.io/fluentbit.key
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : postgresql-libs-9.2.24-7.el7_9.x86_64                                                                                                                                                                                         1/2 
  Installing : td-agent-bit-1.8.2-1.x86_64                                                                                                                                                                                                   2/2 
  Verifying  : postgresql-libs-9.2.24-7.el7_9.x86_64                                                                                                                                                                                         1/2 
  Verifying  : td-agent-bit-1.8.2-1.x86_64                                                                                                                                                                                                   2/2 

Installed:
  td-agent-bit.x86_64 0:1.8.2-1                                                                                                                                                                                                                  

Dependency Installed:
  postgresql-libs.x86_64 0:9.2.24-7.el7_9                                                                                                                                                                                                        

Complete!
[test@node-13 local]$ 

安装完后可以查看安装目录和配置文件位置。

[zyyt@node-13 src]$ rpm -ql td-agent-bit
/etc/td-agent-bit
/etc/td-agent-bit/parsers.conf
/etc/td-agent-bit/plugins.conf
/etc/td-agent-bit/td-agent-bit.conf
/lib/systemd/system/td-agent-bit.service
/lib64/td-agent-bit
/lib64/td-agent-bit/libfluent-bit.so
/opt/td-agent-bit
/opt/td-agent-bit/bin
/opt/td-agent-bit/bin/td-agent-bit

3. 启动关闭

启动 Fluent Bit：

[test@node-13 src]$ ps -ef|grep td-agent-bit
test      68229  61548  0 15:18 pts/2    00:00:00 grep --color=auto td-agent-bit
[test@node-13 src]$ sudo service td-agent-bit start
[sudo] password for test: 
Redirecting to /bin/systemctl start td-agent-bit.service
[test@node-13 src]$ 

查看状态：

[test@node-13 src]$ service td-agent-bit status
Redirecting to /bin/systemctl status td-agent-bit.service
● td-agent-bit.service - TD Agent Bit
   Loaded: loaded (/usr/lib/systemd/system/td-agent-bit.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-21 15:18:29 CST; 35s ago
 Main PID: 68285 (td-agent-bit)
   CGroup: /system.slice/td-agent-bit.service
           └─68285 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
[test@node-13 src]$ ps -ef|grep td-agent-bit
root      68285      1  0 15:18 ?        00:00:00 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
test      68685  61548  0 15:21 pts/2    00:00:00 grep --color=auto td-agent-bit
[test@node-13 src]$ 

查看输出：

[test@node-13 td-agent-bit]$ cd /var/log/
[test@node-13 log]$ sudo tail -f messages
[sudo] password for test:
    下面会输出收集到的日志信息

关闭 Fluent Bit：

[test@node-13 td-agent-bit]$ ps -ef|grep td-agent-bit
root      68285      1  0 15:18 ?        00:00:02 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
test      72227  71946  0 16:15 pts/2    00:00:00 grep --color=auto td-agent-bit
[test@node-13 td-agent-bit]$ sudo service td-agent-bit stop
Redirecting to /bin/systemctl stop td-agent-bit.service
[test@node-13 td-agent-bit]$ ps -ef|grep td-agent-bit
test      72545  71946  0 16:19 pts/2    00:00:00 grep --color=auto td-agent-bit
[test@node-13 td-agent-bit]$ 

4. 配置修改

具体配置参数含义可参考我翻译的官方文档：官方 Fluent Bit 1.8 文档。

主要是修改以下两个文件，parsers.conf配置 fluent bit 的解析器。td-agent-bit.conf是主配置文件。

/etc/td-agent-bit/parsers.conf
/etc/td-agent-bit/td-agent-bit.conf

接下来就是按照业务需求修改这两个配置文件了。这里配置为收集日志到 Kafka。

parsers.conf的主要内容如下，我只添加了一个multiline_match解析器。其他内容都是默认的。

[PARSER]
    Name multiline_match
    Format regex
    Regex /^(?<log_time>\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}.\d+ \+0800)\s+(?<level>[A-Z]+) \[(?<class>[^\]]+)\] \[(?<thread>[^\]]+)\] (?<message>.*)/

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache_error
    Format regex
    Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    # https://rubular.com/r/IhIbCAIs7ImOkc
    Name        k8s-nginx-ingress
    Format      regex
    Regex       ^(?<host>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (\[(?<proxy_alternative_upstream_name>[^ ]*)\] )?(?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<reg_id>[^ ]*).*$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   json
    Format json
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name         docker
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On
    # --
    # Since Fluent Bit v1.2, if you are parsing Docker logs and using
    # the Kubernetes filter, it's not longer required to decode the
    # 'log' key.
    #
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    #Decode_Field_As    json     log

[PARSER]
    Name        docker-daemon
    Format      regex
    Regex       time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

[PARSER]
    Name        syslog-rfc5424
    Format      regex
    Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*?)\]|-)) (?<message>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

[PARSER]
    Name        syslog-rfc3164-local
    Format      regex
    Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
    Time_Key    time
    Time_Format %b %d %H:%M:%S
    Time_Keep   On

[PARSER]
    Name        syslog-rfc3164
    Format      regex
    Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
    Time_Key    time
    Time_Format %b %d %H:%M:%S
    Time_Keep   On

[PARSER]
    Name    mongodb
    Format  regex
    Regex   ^(?<time>[^ ]*)\s+(?<severity>\w)\s+(?<component>[^ ]+)\s+\[(?<context>[^\]]+)]\s+(?<message>.*?) *(?<ms>(\d+))?(:?ms)?$
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On
    Time_Key time

[PARSER]
    # https://rubular.com/r/3fVxCrE5iFiZim
    Name    envoy
    Format  regex
    Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<code>[^ ]*) (?<response_flags>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)"  
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
    Time_Key start_time

[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[PARSER]
    Name    kube-custom
    Format  regex
    Regex   (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

td-agent-bit.conf配置内容如下，将tail收集到的日志发送到kafka。

[SERVICE]
    Flush        5
    Daemon       off
    Log_level     info
    Parsers_file parsers.conf
    Plugins_file  plugins.conf
[INPUT]
    Name               tail
    Tag                azkaban-web
    Path               /home/test/azkaban/azkaban-web-server/logs/azkaban-webserver.log
    Skip_Long_Lines    on
    DB                 /home/data/fluent-bit/azkaban-web.db
    Buffer_Chunk_Size  32k
    Buffer_Max_Size    100k
    Multiline          On
    Parser_Firstline   multiline_match
    Path_Key           log_file_path
    Mem_Buf_Limit      5MB
[OUTPUT]
    Name        kafka
    Match       *
    Brokers     192.168.50.10:9092,192.168.50.11:9092,192.168.50.12:9092
    Topics      fluent-bit-test
    Timestamp_key  timestamp
    Retry_Limit    false
    rdkafka.log.connection.close false
    rdkafka.queue.buffering.max.kbytes 10240
    rdkafka.request.required.acks 1
[FILTER]
    Name record_modifier
    Match *
    Record source azkaban-web