[系统运维]Mac 使用Docker-compose搭建Log4js+ELK日志收集系统

架构服务

文件夹结构

docker-compose.yml

version: '3.7'
services:
  elasticsearch:
    image: elasticsearch:7.6.2
    container_name: elasticsearch
    privileged: true
    user: root
    environment:
      #设置集群名称为elasticsearch
      - cluster.name=elasticsearch 
      #以单一节点模式启动
      - discovery.type=single-node 
      #设置使用jvm内存大小
      - ES_JAVA_OPTS=-Xms512m -Xmx512m 
    volumes:
      - /Users/brook/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins
      - /Users/brook/elk/elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300

  logstash:
    image: logstash:7.6.2
    container_name: logstash
    ports:
       - 4560:4560
    privileged: true
    environment:
      - TZ=Asia/Shanghai
    volumes:
      #挂载logstash的配置文件
      - /Users/brook/elk/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf 
    depends_on:
      - elasticsearch 
    links:
      #可以用es这个域名访问elasticsearch服务
      - elasticsearch:es 
    

  kibana:
    image: kibana:7.6.2
    container_name: kibana
    ports:
        - 5601:5601
    privileged: true
    links:
      #可以用es这个域名访问elasticsearch服务
      - elasticsearch:es 
    depends_on:
      - elasticsearch 
    environment:
      #设置访问elasticsearch的地址
      - elasticsearch.hosts=http://es:9200 

Docker 安装ELK

1. 进入文件夹
   cd /Users/brook/elk/docker
2. 执行
   docker-compose up -d
3. 赋权限
   chmod 777 /Users/brook/elk/elasticsearch/data

设置密码

elasticsearch

1. 进入容器
   docker exec -it elasticsearch bash
2. 修改elasticsearch.yml文件
   vi config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
# 开启安全控制
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

3. 重启elasticsearch
   exit
docker restart elasticsearch
4. 创建密码
   4.1 进入容器
   docker exec -it elasticsearch /bin/bash
   4.2 设置密码-随机生成密码
   elasticsearch-setup-passwords auto
   4.3 设置密码-手动设置密码
   elasticsearch-setup-passwords interactive

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana]: 
Reenter password for [kibana]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

4.4 访问
curl 127.0.0.1:9200 -u elastic:123456

{
  "name" : "a11d93b7881d",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "w1OsXWaPREWO7V_CZAE5mA",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

kibana

1. 进入容器
   docker exec -it kibana bash
2. 修改 kibana.yml 文件
   vi config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "elastic"  # es账号
elasticsearch.password: "123456"   # es密码
i18n.locale: zh-CN # 中文

3. 重启kibana
   docker restart kibana

账号密码

1. elasticsearch
   http://localhost:9200/
   账号：elastic
   密码：123456
2. kibana
   http://localhost:5601/
   账号：elastic
   密码：123456

logstash.conf

input {
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json
  }
}
output {
  
    elasticsearch {
          hosts => "es:9200"
          index => "%{indexname}-logs-%{+YYYY.MM.dd}"
          user => "elastic"
          password => "123456"
    }
  
}

indexname:获取log4js定义的名称

1. 重启logstash
   docker restart logstash

log4js配置使用

1. 项目中安装log4js
   npm insetll log4js
2. 新建logger.ts文件

var log4js = require('log4js');
import { logstashServer } from '@/config';
// logstashServer.host： 127.0.0.1
// logstashServer.port： 4560
if (!logstashServer.host || !logstashServer.port) {
  console.log('ERROR not config logstash_host or logstash_port');
}

const appendersConfig = {
  appenders: {
    console: { type: 'console' },
    logstash: {
      type: 'log4js-logstash-tcp',
      host: logstashServer.host,
      port: parseInt(logstashServer.port),
      fields: {
        appName: 'node-log-stash',
        indexname: 'default',  // 对应logstash中的%{indexname}
      },
    },
  },
  categories: {
    default: { appenders: ['logstash'], level: 'info' },
  },
};

export class Logger {
  // 错误
  static errorLog(code, message) {
    const level = 'error';
    appendersConfig.appenders.logstash.fields.indexname = `yl-app-${level}`; // 替换
    appendersConfig.categories.default.level = level; // 替换
    log4js.configure(appendersConfig);
    const logger = log4js.getLogger('default');
    logger.level = level;
    logger.error({ errorCode: code, message: message });
  }
}

3. 收集log

import express from 'express';
const router: express.Router = express.Router();
import { failRes, resSuccess } from '@/utils/utils';
import _ from 'lodash';
import { Logger } from '@/utils/logger';

router.get('/app/error', async function (req: any, res: express.Response) {
  try {
    Logger.errorLog(23003, '账户密码错误');
    res.send(resSuccess('Error', ''));
  } catch (err) {
    res.send(failRes(err.code, err.message));
  }
});

export default router;

安装elasticsearch-head

1. 打开chrome网上应用商店

2. 搜索elasticsearch-head
   

3. 打开后连接es: http://localhost:9200/