[大数据]filebeat推送消息到kafka-k8s sidecar

完整示例

本示例为：使用filebeat将apisi的日志发送到kafka中

完整配置文件如下：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: apisix
  namespace: apisix
spec:
  replicas: 1
  strategy:
    rollingUpdate:  #由于replicas为3,则整个升级,pod个数在2-4个之间
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: apisix
    spec:
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.3.2
        args: [
          "-c", "/etc/filebeat/filebeat.yml",
          "-e",
        ]
        volumeMounts:
        - name: app-logs
          mountPath: /log
        - name: apisix-filebeat-config
          mountPath: /etc/filebeat/
      - name: apisix
        image: apisix
        command: ["sh","/startup.sh"]
        volumeMounts:
          - name: host-time
            mountPath: /etc/localtime
          - name: app-logs
            mountPath: /home
        env:
          - name: ADMIN_KEY
            value: 9b2498a6db1d11eb9d03fa16486175d7
          - name: BAYMAX_APP_LOG_MNT_DIR
            value: "/home"
          - name: ETCD_ENDPOINT_1
            value: "http://192.24.24.122:2379"
          - name: ETCD_ENDPOINT_2
            value: "http://192.24.24.110:2379"
          - name: ETCD_ENDPOINT_3
            value: "http://192.24.24.111:2379"
          - name: ETCD_PREFIX
            value: /apisix
          - name: ETCD_PWD
            value: root
          - name: ETCD_USER
            value: root
        ports:
          - name: port1
            containerPort: 9080
            protocol: TCP
          - name: port2
            containerPort: 9180
            protocol: TCP
          - name: port3
            containerPort: 9090
            protocol: TCP
        livenessProbe:
          tcpSocket:
            port: 9080
          initialDelaySeconds: 120
          periodSeconds: 30
        resources:
          limits:
            cpu: "2"
            memory: 1.5Gi
          requests:
            cpu: "0.03"
            memory: 1.0Gi
      volumes:
      - name: host-time
        hostPath:
          path: /etc/localtime
      - name: app-logs
        emptyDir: {}
      - name: apisix-filebeat-config
        configMap:
          name: apisix-filebeat-config
      nodeSelector:
        productline:  apisix
---
apiVersion: v1
kind: Service
metadata:
  name: apisix
  namespace: apisix
spec:
  ports:
    - name: port1
      protocol: TCP
      port: 9080
      nodePort: 30141
    - name: port2
      protocol: TCP
      port: 9180
      nodePort: 30142
    - name: port3
      protocol: TCP
      port: 9090
      nodePort: 30143
  selector:
    app: apisix
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-filebeat-config
  namespace: apisix
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: log
      paths:
        - "/log/access.log"
      fields:
        log_source: access-apisix-log
    - type: log
      paths:
        - "/log/error.log"
      fields:
        log_source: error-apisix-log
    output.kafka:
      hosts: ["100.94.17.118:9095","100.94.12.134:9095","100.94.13.18:9095"]
      topic: 'apisix-filebeat-plugin-log'

配置解析

filebeat容器与服务容器共享空间获取日志

1.两个容器挂载emptyDir从而实现数据共享

volumes类型：emptyDir

以下emptyDir接解释参考：https://blog.csdn.net/qq_33591903/article/details/103529274

????????a) emptyDir的生命周期与所属的pod相同。pod删除时，其emptyDir中的数据也会被删除。

????????b) emptyDir类型的volume在pod分配到node上时被创建，kubernetes会在node上自动分配 一个目录，因此无需指定宿主机node上对应的目录文件。

????????c) emptyDir Volume主要用于某些应用程序无需永久保存的临时目录，多个容器的共享目录等。

volumes配置

      volumes:
      - name: host-time
        hostPath:
          path: /etc/localtime
      - name: app-logs
        emptyDir: {}
      - name: apisix-filebeat-config
        configMap:
          name: apisix-filebeat-config

启动filebeat sidecar

      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.3.2
        args: [
          "-c", "/etc/filebeat/filebeat.yml",
          "-e",
        ]
        volumeMounts:
        - name: app-logs
          mountPath: /log
        - name: apisix-filebeat-config
          mountPath: /etc/filebeat/

apisix（服务）挂载emptyDir

        volumeMounts:
          - name: host-time
            mountPath: /etc/localtime
          - name: app-logs
            mountPath: /home

ConfigMap-创建filebeat配置

apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-filebeat-config
  namespace: apisix
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: log
      paths:
        - "/log/access.log"
      fields:
        log_source: access-apisix-log
    - type: log
      paths:
        - "/log/error.log"
      fields:
        log_source: error-apisix-log
    output.kafka:
      hosts: ["100.94.17.118:9095","100.94.12.134:9095","100.94.13.18:9095"]
      topic: 'apisix-filebeat-plugin-log'

inputs中fields来添加一些额外的标签

output.kafka常用参数：hosts、topic