开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> 大数据 -> 监控不携带metrics接口的服务（nginx、mysql、redis） -> 正文阅读

[大数据]监控不携带metrics接口的服务（nginx、mysql、redis）

prometheus的架构图

在这里插入图片描述

监控不携带metrics接口的服务

普罗米修斯的两种监控服务：

? 1.携带metrics接口的服务

? 可以通过metrics接口来获取服务的监控项、监控信息

? 2.不携带metrics接口的服务

? 通过`EndPoint`造一个metrict接口

? PS：metrics（接口）

监控流程：

1、部署exporter，从而创建一个metrics接口

2、部署EndPoints，链接expertor暴露出来的metrics接口

3、部署Service，基于ServiceMonitor使用

4、创建ServiceMonitor，注入promethues

5、测试服务

6、加入grafana，做大屏展示

一、监控nginx

0 、准备测试环境

server {
    listen       80;
    server_name  localhost;

    location /status {
        stub_status;
    }

在这里插入图片描述

Active connections: 当前nginx正在处理的活动连接数.
 
Server accepts handled requests: 
    nginx启动以来总共处理了23个连接
    成功创建23握手(证明中间没有失败的)
    总共处理了177个请求。
 
Reading: nginx读取到客户端的Header信息数.
Writing: nginx返回给客户端的Header信息数.
Waiting: 开启keep-alive的情况下,这个值等于 active – (reading + writing),意思就是nginx已经处理完成,正在等候下一次请求指令的驻留连接。
所以,在访问效率高,请求很快被处理完毕的情况下,Waiting数比较多是正常的.如果reading +writing数较多,则说明并发访问量非常大,正在处理过程中。

1、部署expertor，从而创建一个`metrics接口

`nginx-prometheus-exporter镜像地址： `https://github.com/nginxinc/nginx-prometheus-exporter

依据这个命令造作
$ docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.9.0 -nginx.plus -nginx.scrape-uri=http://<nginx-plus>:8080/api


端口：9113

在这里插入图片描述

部署expertor，从而创建一个metrics接口


1）准备目录
[root@k8s-master1 ~]# mkdir exporter-nginx
[root@k8s-master1 ~]# cd exporter-nginx/
[root@k8s-master1 exporter-nginx]# vim exporter-nginx-deployment.yaml


2）创建服务
创建一个服务，获取并格式化metrics接口数据

cat > exporter-nginx-deployment.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx-prometheus-exporter
spec:
  selector:
    matchLabels:
      app: nginx-prometheus-exporter
      deploy: nginx-exporter
  template:
    metadata:
      labels:
        app: nginx-prometheus-exporter
        deploy: nginx-exporter
    spec:
      containers:
        - name: nginx-exporter
          image: nginx/nginx-prometheus-exporter:0.9.0
          imagePullPolicy: IfNotPresent
          command:
            - "nginx-prometheus-exporter"
            - "-nginx.scrape-uri=http://192.168.15.32/status"   #因为提前已经在192.168.15.32上安装了nginx   ，所以这里填写nginx安装的地址ip
EOF


3）部署服务
[root@k8s-master1 exporter-nginx]# kubectl apply -f exporter-nginx-deployment.yaml 
deployment.apps/name-prometheus-exporter created

创建一个Service，给予prometheus集群获取格式化好了的metrics接口服务

cat > exporter-service.yaml EOF
kind: Service
apiVersion: v1
metadata:
  name: nginx-prometheus-exporter
spec:
  ports:
    - port: 9113          #端口9113
      targetPort: 9113    #容器内部也是9113
      name: nginx-prometheus-exporter
      protocol: TCP
  selector:
    app: nginx-prometheus-exporter
    deploy: nginx-exporter
EOF



[root@k8s-master1 exporter-nginx]# kubectl apply -f exporter-service.yaml


6）查看部署
[root@k8s-master-01 nginx]# kubectl get pods,svc
NAME  READY   STATUS   RESTARTS   AGE
pod/nginx-prometheus-exporter-776f76467-59msl            1/1     Running            0          5m55s

NAME TYPE CLUSTER-IP  EXTERNAL-IP   PORT(S)          AGE
service/nginx-prometheus-exporter   ClusterIP   10.110.27.71    <none>        9113/TCP         2m50s



7）访问测试
[root@k8s-master-01 nginx]# curl 10.110.27.71:9113/metrics
# HELP nginx_connections_accepted Accepted client connections
# TYPE nginx_connections_accepted counter
nginx_connections_accepted 4
# HELP nginx_connections_active Active client connections
# TYPE nginx_connections_active gauge
nginx_connections_active 1
# HELP nginx_connections_handled Handled client connections
# TYPE nginx_connections_handled counter

这就意味着nginx有这个指标了

2、部署Endpoints，链接expertor暴露出来的metrics接口

# 参考上述Service，此处无需重复创建。

[root@k8s-master-01 nginx]# kubectl get endpoints
NAME       ENDPOINTS            AGE
nginx-prometheus-exporter   10.244.1.128:9113    21m

[root@k8s-master-01 nginx]# curl  10.244.1.128:9113/metrics
...
# HELP nginx_http_requests_total Total http requests
# TYPE nginx_http_requests_total counter
nginx_http_requests_total 5
# HELP nginx_up Status of the last metric scrape
# TYPE nginx_up gauge
nginx_up 1
# HELP nginxexporter_build_info Exporter build information
# TYPE nginxexporter_build_info gauge
nginxexporter_build_info{commit="5f88afbd906baae02edfbab4f5715e06d88538a0",date="2021-03-22T20:16:09Z",version="0.9.0"} 1
....


理论上建立一个service就会对应建立一个endpoints

3、部署Service，基于`ServiceMonitor`使用

# 参考上述Service，此处无需重复创建。

cat > exporter-service.yaml EOF
kind: Service
apiVersion: v1
metadata:
  name: nginx-prometheus-exporter
spec:
  ports:
    - port: 9113          #端口9113
      targetPort: 9113    #容器内部也是9113
      name: nginx-prometheus-exporter
      protocol: TCP
  selector:
    app: nginx-prometheus-exporter
    deploy: nginx-exporter
EOF

4、创建ServiceMonitor，注入promethues

cat > service-monitor.yaml <<EOF
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:   #选择service
    app: nginx-prometheus-exporter-servicemonitor
    deploy: nginx-exporter-servicemonitor
  name: nginx-monitor
  namespace: monitoring    #普罗米修斯的命名空间
spec:
  endpoints:
    - interval: 3s
      port: nginx-prometheus-exporter   #用端口或者名字都可以
  selector:
    matchLabels:
      app: nginx-prometheus-exporter
      deploy: nginx-exporter
  namespaceSelector:
    matchNames:
      - "default"
EOF



2）部署查看
[root@k8s-master1 exporter-nginx]# kubectl apply -f service-monitor.yaml
[root@k8s-master1 exporter-nginx]# kubectl get servicemonitor -n monitoring 
NAME                      AGE
nginx-monitor             25s


3）测试（查看prometheus的控制器）
[root@k8s-master-01 nginx]# kubectl get prometheus -n monitoring 
NAME   VERSION   REPLICAS   AGE
k8s    v2.15.2   2          29h

删除重新部署一下
[root@k8s-master-01 opt]# kubectl delete prometheus -n monitoring k8s 
prometheus.monitoring.coreos.com "k8s" deleted

prometheus-k8s.yaml （见附件）
[root@k8s-master-01 opt]# kubectl apply -f prometheus-k8s.yaml 



重启普罗米修斯监控pod(prometheus-k8s-0),载入监控项（见附件）


4）加入grafana，做大屏展示
略~


grafana官网：https://grafana.com/grafana/dashboards/

在这里插入图片描述

附件：

kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
      - name: alertmanager-main-etcd
        namespace: kube-system
        port: etcd
  image: quay.io/prometheus/prometheus:v2.15.2
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.15.2
  secrets:
    - etcd-certs

二、监控redis

https://github.com/oliver006/redis_exporter

在这里插入图片描述

1.部署测试环境

1）编写部署清单

cat > redis-deploy.yaml <EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      k8s: redis-exporter
  template:
    metadata:
      labels:
        k8s: redis-exporter
    spec:
      containers:
        - name: redis
          imagePullPolicy: IfNotPresent
          image: redis
EOF

2）编写Service清单

cat > redis-svc.yaml <<EOF
kind: Service
apiVersion: v1
metadata:
  name: redis-svc
  labels:
    k8s: redis-exporter
spec:
  ports:
    - port: 6379
      targetPort: 6379
      name: redis
  selector:
    k8s: redis-exporter
EOF

3）部署

[root@k8s-master1 exporter-redis]# kubectl apply -f redis-deploy.yaml
[root@k8s-master1 exporter-redis]# kubectl apply -f redis-svc.yaml

2.配置部署exporter

配置部署exporter，从而创建一个metrics接口

1）编写部署清单

cat > redis-exporter-deploy.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-exporter-server
spec:
  selector:
    matchLabels:
      k8s: redis-exporter-server
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9121"
      labels:
        k8s: redis-exporter-server
    spec:
      containers:
        - name: redis
          image: oliver006/redis_exporter
          args:
            - "--redis.addr=redis://redis-svc.default.svc.cluster.local:6379"
          ports:
            - containerPort: 9121
EOF

2）编写svc清单

cat > redis-exporter-svc.yaml <<EOF
kind: Service
apiVersion: v1
metadata:
  name: redis-exporter-server
  labels:
    k8s: redis-exporter-server-svc
spec:
  ports:
    - port: 9121
      targetPort: 9121
      name: redis-exporter
  selector:
    k8s: redis-exporter-server
EOF

3）部署

[root@k8s-master1 exporter-redis]# kubectl apply -f redis-exporter-deploy.yaml

[root@k8s-master-01 redis]# kubectl get pods
NAME                                                 READY   STATUS              RESTARTS   AGE
redis-6b9dd9c556-7p2b2                               1/1     Running             0          4m54s
redis-exporter-server-6f6476f86f-b7kqk               1/1     Running             0          4m1s
    

[root@k8s-master1 exporter-redis]# kubectl apply -f redis-exporter-svc.yaml


[root@k8s-master-01 redis]# kubectl get svc
NAME                        TYPE        CLUSTER-IP      
redis-svc                   ClusterIP   10.102.221.62   <none>        6379/TCP         105m

3.部署Endpoints

部署Endpoints，链接exporter暴露出来的metrics接口

# 参考上述Service，此处无需重复创建。

4.部署Service

部署Service，基于ServiceMonitor使用

# 参考上述Service，此处无需重复创建。

5.创建ServiceMonitor清单

创建ServiceMonitor，注入promethues

1）编写配置清单

cat > redis-monitor.yaml <<EOF
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    app: redis-prometheus-exporter-servicemonitor
    deploy: redis-exporter-servicemonitor
  name: redis-monitor
  namespace: monitoring
spec:
  endpoints:
    - interval: 3s
      port: redis-exporter
  selector:
    matchLabels:
      k8s: redis-exporter-server-svc
  namespaceSelector:
    matchNames:
      - "default"
EOF

2）部署

[root@k8s-master1 exporter-redis]# kubectl apply -f redis-monitor.yaml

redis-monitor             2m

6.grafana展示

在这里插入图片描述

三、监控数据库

1.造环境

1）安装mariadb

[root@k8s-master1]# yum install mariadb\* -y
[root@k8s-master1]# systemctl enable --now mariadb

2）授权monitor用户

[root@k8s-master1]# mysql
MariaDB [(none)]> grant select,replication client,process ON *.* to 'mysql_monitor'@'%' identified by 'oldboy@123';
Query OK, 0 rows affected (0.00 sec)

PS：授权ip为localhost，因为不是prometheus服务器来直接找mariadb 获取数据，而是prometheus服务器找mysql_exporter,mysql_exporter 再找mariadb，所以这个localhost是指的mysql_exporter的IP

2.配置maridb

创建一个mariadb配置文件，写上连接的用户名与密码(和上面的授权的用户名和密码要对应)

[root@k8s-master1]# vim /usr/local/mysqld_exporter-0.12.1/.my.cnf
[client]
user=mysql_monitor
password=123

3.启动mysqld_exporter

[root@k8s-master1]# nohup /usr/local/mysqld_exporter-0.12.1/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter-0.12.1/.my.cnf &

PS：确认端口是否为 9104

2.安装数据库exporter

1）编写部署清单

cat > mysql-deploy.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: mysql-exporter-server
spec:
  selector:
    matchLabels:
      k8s: mysql-exporter-server
  template:
    metadata:
      labels:
        k8s: mysql-exporter-server
    spec:
      containers:
        - name: mysql
          image: prom/mysqld-exporter
          env:
            - name: DATA_SOURCE_NAME
              value: "mysql_monitor:oldboy@123@(mysql.default.svc.cluster.local:3306)"
          ports:
            - containerPort: 9104
EOF

2）编写Service清单

cat > mysql-svc.yaml <<EOF
kind: Service
apiVersion: v1
metadata:
  name: mysql-exporter-server
  labels:
    k8s: mysql-exporter-server-svc
spec:
  ports:
    - port: 9104
      targetPort: 9104
      name: mysql-exporter-port
      protocol: TCP
  selector:
    k8s: mysql-exporter-server
---
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    app: mysql-prometheus-exporter-servicemonitor
    deploy: mysql-exporter-servicemonitor
  name: mysql-exporter-monitor
  namespace: monitoring
spec:
  endpoints:
    - interval: 3s
      port: mysql-exporter-port
  selector:
    matchLabels:
      k8s: mysql-exporter-server-svc
  namespaceSelector:
    matchNames:
      - "default"
EOF

3.部署服务

[root@k8s-master1 exporter-mysql]# kubectl apply -f mysql-deploy.yaml
[root@k8s-master1 exporter-mysql]# kubectl apply -f mysql-svc.yaml

监控exporter

监控一个外部的exporter

1.造环境

1）下载部署exporter

#下载
[root@k8s-master1]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz

# 解压
[root@k8s-master1]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/

#查看
[root@k8s-master1]# ll /usr/localmysqld_exporter-0.12.1.linux-amd64
总用量 14484
-rw-r--r-- 1 3434 3434    11325 7月  29 2019 LICENSE
-rwxr-xr-x 1 3434 3434 14813452 7月  29 2019 mysqld_exporter
-rw-r--r-- 1 3434 3434       65 7月  29 2019 NOTICE