[系统运维]Pormetheus（普罗米修斯） 采用blackbox_exporter实现监控

简介： BlackBox Exporter 顾名思义就是在应用程序的外部对其进行探测， 支持 HTTP、HTTPS、DNS、TCP、ICMP等方式对目标进行检测。

BlackBox Exporter 顾名思义就是在应用程序的外部对其进行探测，
支持 HTTP、HTTPS、DNS、TCP、ICMP等方式对目标进行检测。

官网文档：https://github.com/prometheus/blackbox_exporter

1、 HTTP 测试

定义 Request Header 信息
判断 Http status / Http Respones Header / Http Body 内容

2、 TCP 测试

业务组件端口状态监听
应用层协议定义与监听

3、 ICMP 测试

主机探活机制

4、 POST 测试

接口联通性

5、 SSL 证书过期时间

运行Blackbox Exporter时，需要用户提供探针的配置信息，这些配置信息可能是一些自定义的HTTP头信息，也可能是探测时需要的一些TSL配置，也可能是探针本身的验证行为。在Blackbox Exporter每一个探针配置称为一个module，并且以YAML配置文件的形式提供给Blackbox Exporter。每一个module主要包含以下配置内容，包括探针类型（prober）、验证访问超时时间（timeout）、以及当前探针的具体配置项：

  # 探针类型：http、 tcp、 dns、 icmp.
  prober: <prober_string>
  
  # 超时时间
  [ timeout: <duration> ]
  
  # 探针的详细配置，最多只能配置其中的一个
  [ http: <http_probe> ]
  [ tcp: <tcp_probe> ]
  [ dns: <dns_probe> ]
  [ icmp: <icmp_probe> ]

2. black_exporter 安装

2.1 docker 方式安装 推荐

docker pull prom/blackbox-exporter

docker run -d -p 9115:9115 --name blackbox-exporter  prom/blackbox-exporter

2.2 宿主机安装
下载地址 https://github.com/prometheus/blackbox_exporter/releases

# 下载
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.18.0/blackbox_exporter-0.19.0.darwin-amd64.tar.gz
# 解压
tar -zxvf blackbox_exporter-0.19.0.darwin-amd64.tar.gz
# 重命名
mv blackbox_exporter-0.19.0.darwin-amd64/ blackbox_exporter
# 进入 blackbox_exporter 目录
cd blackbox_exporter
# 启动
nohup ./blackbox_exporter --web.listen-address=:9115 --config.file=/home/apps/blackbox_exporter/blackbox.yml &

默认监听9115端口：

# ss -tunlp|grep 9115
tcp    LISTEN     0      32768                  *:9115                  *:*      users:(("blackbox_export",29880,3))

blackbox.yml 配置文件介绍，具体配置参考https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md：

modules:
  http_2xx: # http监测模块，这个名字是随便写的，但是需要在 prometheus.yml 配置文件中对应起来。
    prober: http # 进行探测的协议，可以是 http、tcp、dns、icmp
    timeout: 10s
  http_post_2xx:  # http post检测模块
    prober: http
    http:
      method: POST
  tcp_connect: # tcp检测，比如检测某个端口是否在线
    prober: tcp
  icmp: # icmp 检测，比如ping某个服务器
    prober: icmp

3.监控主机存活状态

# vim prometheus.yml
- job_name: crawler_status
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['10.45.3.8','10.45.3.9','10.45.3.10','10.45.3.11']
        labels:
          instance: node_status
          group: 'node'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

注意：10.45.3.8是被监控端ip，127.0.0.1是Blackbox_exporter

3.监控主机端口存活状态

- job_name: 'crawler_port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['10.45.3.8:3128']
        labels:
          instance: 'port_status'
          group: 'tcp'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

4.监控网站状态 URL

- job_name: qianxin_web_status
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.qianxin.com']
        labels:
          instance: qianxin_web_status
          group: 'web'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115

- job_name: baidu_web_status
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.baidu.com']
        labels:
          instance: baidu_web_status
          group: 'web'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115

- job_name: jd_web_status
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.jd.com']
        labels:
          instance: jd_web_status
          group: 'web'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115

URL部分详解：

root@prometheus-01:~# cat /usr/local/prometheus/prometheus.yml 

#网站监控
  - job_name: 'http_status'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:                  # 宿主机或docker不开启IPV6，则不支持IPV6解析。
        - http://www.baidu.com    # Target to probe with http.
        - https://www.xiaomi.com   # Target to probe with https.
        - http://192.168.174.120:9090 # Target to probe with http on port 8080.
        labels:
          instance: http_status
          group: web
    relabel_configs:
      - source_labels: [__address__]  #relabel 通过将__address__(当前目标地址)写入__param_target标签来创建一个label
        target_label: __param_target #监控目标 www.baidu.com，作为__address__的value
      - source_labels: [__param_target] #监控目标
        target_label: url #将监控目标与URL创建一个label
      - target_label: __address__
        replacement: 192.168.174.104:9115  # The blackbox exporter's real hostname:port.

完整测试配置可参考：

global:
  scrape_interval:     60s
  evaluation_interval: 60s
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
 
  - job_name: linux
    static_configs:
      - targets: ['172.19.209.51:9100']
        labels:
          instance: localhost

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
       module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://www.baidu.com
        labels:
          instance: www.baidu.com
          group: web

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        #target_label: instance
        target_label: url
      - target_label: __address__
        replacement: 172.19.209.51:9115  # The blackbox exporter's real hostname:port.

  - job_name: 'blackbox-01'
    metrics_path: /probe
    params:
       module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://www.cc.com
        labels:
          instance: www.cc.com
          group: web

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: url
      - target_label: __address__
        replacement: 172.19.209.51:9115  # The blackbox exporter's real hostname:port.

5.检查配置文件是否书写正确

#/usr/local/prometheus/promtool check config /usr/local/prometheus/prometheus.yml 
Checking /usr/local/prometheus/prometheus.yml
  SUCCESS: 1 rule files found

Checking /usr/local/prometheus/alert_rules/instance_up.yaml
  SUCCESS: 1 rules found

6.重启prometheus

systemctl restart prometheus
或
docker restart ID

7.查看targets界面

8.grafana中加入blackbox_exporter监控数据

安装grafana

docker run -d -p 3000:3000 --name grafana grafana/grafana

运行完成打开,127.0.0.1:3000

登录页面输入 超级管理员的账号密码

第一次登录 默认 账号 admin 密码admin

2、添加数据源

9.grafana中加入blackbox_exporter监控数据

1、添加模版

https://grafana.com/grafana/dashboards/9965
https://grafana.com/grafana/dashboards/9719

此模板需要安装饼状图插件 下载地址 https://grafana.com/grafana/plugins/grafana-piechart-panel
安装插件，重启grafana生效。

docker exec -it grafana grafana-cli plugins install grafana-piechart-panel
docker restart grafana

效果图： 推荐：11543

2022深度学习开发者峰会

5月20日13:00让我们相聚云端，共襄盛会！