[系统运维]loki+promtail+grafana日志框架搭建

下载loki promtail grafana

#loki promtail 
https://github.com/grafana/loki/releases
#grafana
https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1

启动命令

./grafana-server web &
nohup /mnt/data/filebeat/filebeat -e > logs/stdout.log 2>&1 &
?
nohup ./loki-linux-amd64 -config.file=/mnt/loki-local-config.yaml > log-loki.file 2>&1 &
nohup ./promtail -config.file=/mnt/data/promtail-config.yaml > log-promtail.file 2>&1 &

loki配置文件

auth_enabled: false
?
server:
  http_listen_port: 8100
?
ingester:
  lifecycler:
 ?  address: 127.0.0.1
 ?  ring:
 ? ?  kvstore:
 ? ? ?  store: inmemory
 ? ?  replication_factor: 1
 ?  final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0
?
schema_config:
  configs:
 ?  - from: 2020-06-22
 ? ?  store: boltdb
 ? ?  object_store: filesystem
 ? ?  schema: v11
 ? ?  index:
 ? ? ?  prefix: index_
 ? ? ?  period: 168h
?
storage_config:
  boltdb:
 ?  directory: /mnt/data/ELK/loki/data/index
?
  filesystem:
 ?  directory: /mnt/data/ELK/loki/data/chunks
?
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
?
chunk_store_config:
  max_look_back_period: 0s
?
table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

promtail配置文件

server:
  http_listen_port: 9080
  grpc_listen_port: 0
?
positions:
  filename: /tmp/positions.yaml
?
clients:
  - url: http://192.168.13.82:8100/loki/api/v1/push
?
scrape_configs:
- job_name: 28-home-log
  static_configs:
  - targets:
 ? ?  - localhost
 ?  labels:
 ? ?  job: 28-home-log
 ? ?  host: 192.168.13.28
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-home/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.29
 ?  labels:
 ? ?  job: 29-home-log
 ? ?  host: 192.168.13.29
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-home/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.29
 ?  labels:
 ? ?  job: 29-smart-log
 ? ?  host: 192.168.13.29
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-smart/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.30
 ?  labels:
 ? ?  job: 30-smart-log
 ? ?  host: 192.168.13.30
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-smart/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.42
 ?  labels:
 ? ?  job: 42-smart-log
 ? ?  host: 192.168.13.42
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-smart/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.42
 ?  labels:
 ? ?  job: 42-home-log
 ? ?  host: 192.168.13.42
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-home/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.75
 ?  labels:
 ? ?  job: 75-smart-log
 ? ?  host: 192.168.13.75
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-smart/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.75
 ?  labels:
 ? ?  job: 75-home-log
 ? ?  host: 192.168.13.75
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-home/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.77
 ?  labels:
 ? ?  job: 77-smart-log
 ? ?  host: 192.168.13.77
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-smart/*/logs/*.log
?
  - targets:
 ? ?  - 192.168.13.78
 ?  labels:
 ? ?  job: 78-home-log
 ? ?  host: 192.168.13.78
 ? ?  env: prod
 ? ?  __path__: /mnt/data/iot-home/*/logs/*.log

?
?

启停脚本

#!/bin/bash
?
start(){
 ?  pid=`ps -ef |grep promtail |grep -v grep |awk '{print $2}'`
 ?  if [ ! "$pid" ];then
 nohup ./promtail -config.file=/mnt/data/promtail/promtail-config.yaml > promtail.log 2>&1 &
 echo "Starting promtail success!"
 ?  else
 ? ? ?  echo "promtail is already start!"
 ? ? ?  echo -n "pid : $pid"
 ? ? ?  exit
 ?  fi
}
?
stop(){
 ?  echo -n $"Stopping promtail: "
 ?  pid=`ps -ef |grep promtail |grep -v grep |awk '{print $2}'`
 ?  if [ ! "$pid" ];then
  echo "promtail is not running!"
 ?  else
 echo "$pid"
 ? ? ?  kill $pid
 echo "stop promtail success! pid : $pid"
 ?  fi
}
?
restart() {
 ?  stop
 ?  sleep 3s
 ?  start
}
?
status(){
 ?  pid=`ps -ef |grep promtail |grep -v grep |awk '{print $2}'`
 ?  if [ ! "$pid" ];then
 ? ? ?  echo "promtail is not running"
 ?  else
 ? ? ?  echo "promtail is running! pid: $pid"
 ?  fi
}
?
case "$1" in
 ?  start)
 ? ? ?  start
 ?  ;;
 ?  stop)
 ? ? ?  stop
 ?  ;;
 ?  restart)
 ? ? ?  restart
 ?  ;;
 ?  status)
 ? ? ?  status
 ?  ;;
 ?  *)
 ? ? ?  echo $"Usage: $0 {start|stop|restart|status}"
 ? ? ?  exit 1
esac

grafana配置

1.添加loki数据源

?

2.打开展示板 explore

?

3.查看日志数据 选择标签 可以在配置文件自定义标签

?

4.查询语法

?

Loki Cheat Sheet See your logs Start by selecting a log stream from the Log labels selector. Alternatively, you can write a stream selector into the query field: {job="default/prometheus"} Combine stream selectors {app="cassandra",namespace="prod"} Returns all log lines from streams that have both labels. Filtering for search terms. {app="cassandra"} |~ "(duration|latency)s*(=|is|of)s*[d.]+" {app="cassandra"} |= "exact match" {app="cassandra"} != "do not match" LogQL supports exact and regular expression filters. Count over time count_over_time({job="mysql"}[5m]) This query counts all the log lines within the last five minutes for the MySQL job. Rate rate(({job="mysql"} |= "error" != "timeout")[10s]) This query gets the per-second rate of all non-timeout errors within the last ten seconds for the MySQL job. Aggregate, count, and group sum(count_over_time({job="mysql"}[5m])) by (level) Get the count of logs during the last five minutes, grouping by level