[大数据] 高可用pxc+rocketmq+es+redis+minio+keepalive+haproxy 实操

开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> 大数据 -> 高可用pxc+rocketmq+es+redis+minio+keepalive+haproxy 实操 -> 正文阅读

[大数据]高可用pxc+rocketmq+es+redis+minio+keepalive+haproxy 实操

文章很长，建议收藏起来慢慢读！ 总目录博客园版 为您奉上珍贵的学习资源：

免费赠送 :《尼恩Java面试宝典》持续更新+ 史上最全 + 面试必备 2000页+ 面试必备 + 大厂必备 +涨薪必备

免费赠送经典图书:《Java高并发核心编程（卷1）》面试必备 + 大厂必备 +涨薪必备 加尼恩免费领
免费赠送经典图书:《Java高并发核心编程（卷2）》面试必备 + 大厂必备 +涨薪必备 加尼恩免费领
免费赠送经典图书:《Netty Zookeeper Redis 高并发实战》面试必备 + 大厂必备 +涨薪必备 加尼恩免费领
免费赠送经典图书:《SpringCloud Nginx高并发核心编程》面试必备 + 大厂必备 +涨薪必备 加尼恩免费领
免费赠送资源宝库： Java 必备百度网盘资源大合集价值>10000元 加尼恩领取

推荐：尼恩Java面试宝典（史上最全 + 面试必备）详请点击此链接

尼恩Java面试宝典，33个最新pdf，含2000多页，不断更新、持续迭代 具体详情，请点击此链接
在这里插入图片描述

Docker Swarm

Docker Swarm 是 Docker 官方三剑客项目之一，提供 Docker 容器集群服务，是 Docker 官方对容器云生态进行支持的核心方案。使用它，用户可以将多个 Docker 主机封装为单个大型的虚拟 Docker 主机，快速打造一套容器云平台。

Docker 1.12 Swarm mode 已经内嵌入 Docker 引擎，成为了 docker 子命令 docker swarm。请注意与旧的 Docker Swarm 区分开来。

Swarm mode内置kv存储功能，提供了众多的新特性，比如：具有容错能力的去中心化设计、内置服务发现、负载均衡、路由网格、动态伸缩、滚动更新、安全传输等。使得 Docker 原生的 Swarm 集群具备与 Mesos、Kubernetes 竞争的实力。

基本概念

Swarm 是使用 SwarmKit 构建的 Docker 引擎内置（原生）的集群管理和编排工具。

节点：
运行 Docker 的主机可以主动初始化一个 Swarm 集群或者加入一个已存在的 Swarm 集群，这样这个运行 Docker 的主机就成为一个 Swarm 集群的节点 (node)

节点分为管理 (manager) 节点和工作 (worker) 节点

管理节点：用于 Swarm 集群的管理，docker swarm 命令基本只能在管理节点执行（节点退出集群命令 docker swarm leave 可以在工作节点执行）。一个 Swarm 集群可以有多个管理节点，但只有一个管理节点可以成为 leader，leader 通过 raft 协议实现。
工作节点：是任务执行节点，管理节点将服务 (service) 下发至工作节点执行。管理节点默认也作为工作节点。也可以通过配置让服务只运行在管理节点。

集群中管理节点与工作节点的关系如下所示：
这里写图片描述
服务和任务：
任务（Task）是 Swarm 中的最小的调度单位，目前来说就是一个单一的容器。
服务（Services）是指一组任务的集合，服务定义了任务的属性。服务有两种模式：

replicated services 按照一定规则在各个工作节点上运行指定个数的任务。
global services 每个工作节点上运行一个任务。

两种模式通过 docker service create 的 --mode 参数指定。
容器、任务、服务的关系如下所示：
这里写图片描述

swarm：
从 v1.12 开始，集群管理和编排功能已经集成进 Docker Engine。

当 Docker Engine 初始化了一个 swarm 或者加入到一个存在的 swarm 时，它就启动了 swarm mode。没启动 swarm mode 时，Docker 执行的是容器命令；运行 swarm mode 后，Docker 增加了编排 service 的能力。
Docker 允许在同一个 Docker 主机上既运行 swarm service，又运行单独的容器。

node：
swarm 中的每个 Docker Engine 都是一个 node，有两种类型的 node：manager 和 worker。
为了向 swarm 中部署应用，我们需要在 manager node 上执行部署命令，manager node 会将部署任务拆解并分配给一个或多个 worker node 完成部署。

manager node 负责执行编排和集群管理工作，保持并维护 swarm 处于期望的状态。swarm 中如果有多个 manager node，它们会自动协商并选举出一个 leader 执行编排任务。

woker node 接受并执行由 manager node 派发的任务。默认配置下 manager node 同时也是一个 worker node，不过可以将其配置成 manager-only node，让其专职负责编排和集群管理工作。work node 会定期向 manager node 报告自己的状态和它正在执行的任务的状态，这样 manager 就可以维护整个集群的状态。

service：
service 定义了 worker node 上要执行的任务。

swarm 的主要编排任务就是保证 service 处于期望的状态下。
举一个 service 的例子：在 swarm 中启动一个 http 服务，使用的镜像是 httpd:latest，副本数为 3。manager node 负责创建这个 service，经过分析知道需要启动 3 个 httpd 容器，根据当前各 worker node 的状态将运行容器的任务分配下去，比如 worker1 上运行两个容器，worker2 上运行一个容器。运行了一段时间，worker2 突然宕机了，manager 监控到这个故障，于是立即在 worker3 上启动了一个新的 httpd 容器。这样就保证了 service 处于期望的三个副本状态。

默认配置下 manager node 也是 worker node，所以 swarm-manager 上也运行了副本。如果不希望在 manager 上运行 service，可以执行如下命令：

docker node update --availability drain master

应用案例

集群最擅长的就是解决多服务问题，只要在同一 network 之下，服务之间默认可以直接通过 service_name 互通有无。但为了避免混乱，各服务与外部的通信最好统一交给一个代理服务转发。因对 nginx 比较熟悉，所以我最初选择的代理是“jwilder/nginx-proxy”：

部署架构

众所周知，compose 就是 docker提供给我们用来进行编排的利器，那么我们应该如何使用她呢？

在解析docker-compose.yml之前，我想先说明一下我的业务，在熟悉业务之后，我们再来看看具体的"code".

业务需求：我们要在两个host (swarm node)上面分别建立一组服务，为了简单起见，我将nodes命名为nodeDB和nodeService；顾名思义，我们就是要在nodeDB上面建立database的服务（我这里是部署oracle server），在nodeService上面部署我们的程序(containers). 而 containers 之间的通信就是通过Swarm的Overlay网络来完成的。

准备工作

首先是通过 Swarm 创建 nodes,并且创建Overlay网络模式.

Docker的原生overlay网络的实现原理

Docker版本信息
manager node: Docker version 18.09.4, build d14af54266

worker node: Docker version 19.03.1, build 74b1e89

Docker Swarm系统环境
manager node: 192.168.246.194

worker node: 192.168.246.195

创建 docker swarm集群前的网络
manager node:

docker network ls

NETWOrk ID NAME DRIVER SCOPE
e01d59fe00e5 bridge bridge local
15426f623c37 host host local
dd5d570ac60e none null local

worker node:

docker network ls

NETWOrk ID NAME DRIVER SCOPE
70ed15a24acd bridge bridge local
e2da5d928935 host host local
a7dbda3b96e8 none null local

创建 docker swarm 集群

初始化 docker swarm 集群

manager node执行：

docker swarm init

执行结果如下

[root@localhost ~]#  docker swarm init
Swarm initialized: current node (45mgl95ewnkovrl62nzcgg39i) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-58420uvva5gq6muv4r8swp2ftojhccwdiss9in0jhsagqdk8cm-9c193wz4ye2sucdxv2enofmwh 172.18.8.101:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

初始化swarm或将Docker主机加入现有swarm时，会在该Docker主机上创建两个新网络：

名称为ingress的overlay网络，处理与swarm集群服务相关的控制和数据流量。swarm群组服务并不会将其连接到用户定义的overlay网络时，而是连接到默认的ingress网络，若要连接到用户自定义的overlay网络需要使用–network指定(但创建该用户自定义的overlay网络时必须使用–attachable选项)。
名为docker_gwbridge的桥接网络，它将参与到该swarm群集的各个Docker守护程序连接连接起来。

加入 docker swarm 集群

worker node执行:

docker swarm join --token SWMTKN-1-58420uvva5gq6muv4r8swp2ftojhccwdiss9in0jhsagqdk8cm-9c193wz4ye2sucdxv2enofmwh 172.18.8.101:2377

node2 执行结果如下

[root@localhost ~]# docker swarm join --token SWMTKN-1-58420uvva5gq6muv4r8swp2ftojhccwdiss9in0jhsagqdk8cm-9c193wz4ye2sucdxv2enofmwh 172.18.8.101:2377
This node joined a swarm as a worker.

node3 执行结果如下

[root@k8s-master ~]# docker swarm join --token SWMTKN-1-58420uvva5gq6muv4r8swp2ftojhccwdiss9in0jhsagqdk8cm-9c193wz4ye2sucdxv2enofmwh 172.18.8.101:2377
This node joined a swarm as a worker.

查看集群节点信息

manager node 执行:

docker node ls

manager node 执行结果如下

[root@localhost ~]# docker node ls
ID                            HOSTNAME                STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
ijhuv473dmmfo518szfryaiz4     k8s-master              Ready               Active                                  19.03.6
45mgl95ewnkovrl62nzcgg39i *   localhost.localdomain   Ready               Active              Leader              19.03.6
e2v9g0f7fvp448amvtcwbh2ur     localhost.localdomain   Ready               Active                                  19.03.6

查看集群网络信息

manager node 执行:

docker network ls

manager node 执行结果如下

[root@localhost ~]# docker network ls
NETWORK ID          NAME                  DRIVER              SCOPE
d640a4069a2c        bridge                bridge              local
769df470937b        docker_gwbridge       bridge              local
b95c88a0643f        flowable-ui_default   bridge              local
e86bdbb89c2a        host                  host                local
fgw2irojkxus        ingress               overlay             swarm
d2d2adc84539        kafka_default         bridge              local
5963708cf3f5        mysql5733_default     bridge              local
70a8f84aba79        nacos132_default      bridge              local
b5c13b4d6f08        none                  null                local
7a07f786100b        pg96_default          bridge              local
c6fba23185fd        pg114_default         bridge              local
2a7d23e666cb        redis608_default      bridge              local

worker node执行:

docker network ls

执行结果如下

[root@localhost ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
f7c823f5555a        bridge              bridge              local
5d99c0e15044        docker_gwbridge     bridge              local
994c662e2c74        host                host                local
fgw2irojkxus        ingress             overlay             swarm
7286f698b35f        kafka_default       bridge              local
96a79b1a8e68        nacos_default       bridge              local
4797f82b5944        none                null                local
cf17d1eaa60f        rmq                 bridge              local

查看节点的网络空间

在docker swarm集群创建的开始，docker 会给每台host创建除了docker0以外的两个网络，分是bridge类型(docker_gwbridge网桥)和overlay类型(ingress)的网络，

docker 会创建一个过渡的命名空间ingress_sbox，如下是 node2 上的网络命名空间：

[root@k8s-master ~]#  ip netns ls
[root@k8s-master ~]# cd /run/docker/netns/
[root@k8s-master netns]# ls
02a9055f4fd8  1885e369908f  21ed1da98c33  437047e9edd6  85492903d17c  a3aff3ee377d  bbe1ade3a6ee  default
05eea38af954  1-fgw2irojkx  28e7ea276af9  63f184ecf905  8de9ada0c101  a4a7b1a4fc59  c7f2929f0121  ingress_sbox
0609a84c2a7d  21e63d02ea43  37e6ab5abee1  7c1e39933366  986c4f4289b7  ac8c98ffd1e4  c906d37da694

manager节点自建overlay网络

创建用户自定义的用于swarm服务的overlay网络，使用如下命令：

docker network create -d overlay my-overlay

创建可由swarm群集服务或独立容器用于与在其他Docker守护程序上运行的其他独立容器通信的overlay网络，请添加–attachable标志：

docker network create -d overlay --attachable my-attachable-overlay

可以使用如下命令在 manager节点自建overlay网络，结果如下：

docker network create -d overlay --attachable ha-network-overlay

执行结果如下

[root@localhost ~]# docker network create -d overlay ha_network
kj0qk79lkuhcvz7ovrp71c6jo

再次查看 manager 和 worker 两台主机 docker swarm 集群网络：

manager node:

[root@localhost ~]# docker network ls
NETWORK ID          NAME                  DRIVER              SCOPE
d640a4069a2c        bridge                bridge              local
769df470937b        docker_gwbridge       bridge              local
b95c88a0643f        flowable-ui_default   bridge              local
7x41wgxztm8z        ha-network-overlay    overlay             swarm
kj0qk79lkuhc        ha_network            overlay             swarm
e86bdbb89c2a        host                  host                local
fgw2irojkxus        ingress               overlay             swarm
d2d2adc84539        kafka_default         bridge              local
5963708cf3f5        mysql5733_default     bridge              local
70a8f84aba79        nacos132_default      bridge              local
b5c13b4d6f08        none                  null                local
7a07f786100b        pg96_default          bridge              local
c6fba23185fd        pg114_default         bridge              local
2a7d23e666cb        redis608_default      bridge              local

worker node:

[root@localhost ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
f7c823f5555a        bridge              bridge              local
5d99c0e15044        docker_gwbridge     bridge              local
994c662e2c74        host                host                local
fgw2irojkxus        ingress             overlay             swarm
7286f698b35f        kafka_default       bridge              local
96a79b1a8e68        nacos_default       bridge              local
4797f82b5944        none                null                local
cf17d1eaa60f        rmq                 bridge              local

我们会发现在 worker node上并没有ha_network 网络。

这是因为只有当运行中的容器连接到覆盖网络的时候，该网络才变为可用状态。这种延迟生效策略通过减少网络梳理，提升了网络的扩展性。

先决条件

创建一个overlay网络Prerequisites:先决条件

你需要开放以下端口，以便和参与overlay网络的每个Docker主机进行通信：

TCP port 2377 for cluster management communications（用于群集管理通信的TCP端口2377）
TCP and UDP port 7946 for communication among nodes（用于节点间通信的TCP和UDP端口7946）
UDP port 4789 for overlay network traffic（overlay网络流量的UDP端口4789）

在创建覆盖网络之前，您需要使用docker swarm init将Docker守护程序初始化为swarm manager，或者使用docker swarm join将其连接到现有的swarm。

其中任何一个都会创建默认的ingress overlay网络， swarm服务默认使用该网络。即使你从未打算使用swarm服务，你也需要这么做。之后，可以创建其他用户定义的覆盖网络。

在work节点让网络生效

如果网络没有生效，直接运行"docker-compose up"会报错，内容大致为找不到指定的网络。。。

这是因为只有当运行中的容器连接到覆盖网络的时候，该网络才变为可用状态。

所以我们需要先在 worknode 上面运行一个container让其加入网络 “ha_network “

例如

scp /root/ha-nginx.tar root@172.18.8.104:/root

docker load -i /root/ha-nginx.tar

docker run -itd --name=test-ha-nginx --network=ha-network-overlay custom/ha-nginx:1.0 /bin/sh

输入结果如下：

[root@localhost ~]# docker stop abb54723b8b3a09c7103edba84e4341a2bd9a45ac902c1c098b98a22c3c2187d
abb54723b8b3a09c7103edba84e4341a2bd9a45ac902c1c098b98a22c3c2187d
[root@localhost ~]# docker rm abb54723b8b3a09c7103edba84e4341a2bd9a45ac902c1c098b98a22c3c2187d
abb54723b8b3a09c7103edba84e4341a2bd9a45ac902c1c098b98a22c3c2187d
[root@localhost ~]# docker run -itd --name=test-ha-nginx --network=ha-network-overlay custom/ha-nginx:1.0  /bin/sh
fd4d197d4e111a1caf64cbda5e09d82ecbd2858a57ccb0ca80fd36f7e26d7e56

worker node看网络：

[root@localhost ~]# docker network ls
NETWORK ID          NAME                 DRIVER              SCOPE
f7c823f5555a        bridge               bridge              local
5d99c0e15044        docker_gwbridge      bridge              local
7x41wgxztm8z        ha-network-overlay   overlay             swarm
994c662e2c74        host                 host                local
fgw2irojkxus        ingress              overlay             swarm
7286f698b35f        kafka_default        bridge              local
96a79b1a8e68        nacos_default        bridge              local
4797f82b5944        none                 null                local
cf17d1eaa60f        rmq                  bridge              local

在work节点让 ha-network-overlay 网络生效

查看网络命名空间

由于容器和overlay的网络的网络命名空间文件不再操作系统默认的/var/run/netns下，在/var/run/docker/netns 下，

[root@localhost ~]# ll /var/run/docker/netns
total 0
-r--r--r-- 1 root root 0 Sep  9 19:22 019a382e24bc
-r--r--r-- 1 root root 0 Nov  2 11:47 1-7x41wgxztm
-r--r--r-- 1 root root 0 Nov  2 10:12 1-fgw2irojkx
-r--r--r-- 1 root root 0 Sep  9 19:23 316692258e98
-r--r--r-- 1 root root 0 Sep  9 19:23 326000d15579
-r--r--r-- 1 root root 0 Sep  9 19:23 3e27e6ae976d
-r--r--r-- 1 root root 0 Sep  9 19:22 40a3d5256892
-r--r--r-- 1 root root 0 Sep  9 19:22 60eff8e6cd57
-r--r--r-- 1 root root 0 Sep  9 19:22 80d2ed3639de
-r--r--r-- 1 root root 0 Nov  2 11:47 97748869b254
-r--r--r-- 1 root root 0 Sep  9 19:23 b0464dcf7025
-r--r--r-- 1 root root 0 Apr 27  2021 default
-r--r--r-- 1 root root 0 Nov  2 10:12 ingress_sbox
-r--r--r-- 1 root root 0 Nov  2 11:47 lb_7x41wgxzt

只能手动通过软连接的方式查看。

ln -s /var/run/docker/netns /var/run/netns

master node:

[root@localhost ~]# ln -s /var/run/docker/netns  /var/run/netns
[root@localhost ~]# ip netns
1-fgw2irojkx (id: 5)
ingress_sbox (id: 6)
ab71193396d7 (id: 4)
e63422517d09 (id: 3)
7d3903712aea (id: 0)
8902a33e8595 (id: 2)
227fbb39c26d (id: 1)

worker node:

[root@localhost ~]# ln -s /var/run/docker/netns  /var/run/netns
[root@localhost ~]# ip netns
97748869b254 (id: 12)
1-7x41wgxztm (id: 10)
lb_7x41wgxzt (id: 11)
1-fgw2irojkx (id: 8)
ingress_sbox (id: 9)
316692258e98 (id: 0)
3e27e6ae976d (id: 4)
b0464dcf7025 (id: 5)
326000d15579 (id: 2)
60eff8e6cd57 (id: 3)
019a382e24bc (id: 7)
80d2ed3639de (id: 6)
40a3d5256892 (id: 1)
default

有时候网络的网络命名空间名称前面会带上1-、2-等序号，有时候不带。但不影响网络的通信和操作。

查看网络信息

docker network inspect  ha-network-overlay

master node:

[root@localhost ~]#  docker network inspect  ha-network-overlay
[
    {
        "Name": "ha-network-overlay",
        "Id": "7x41wgxztm8zs7n4gslsfdd80",
        "Created": "2021-11-02T03:42:57.663035559Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.3.0/24",
                    "Gateway": "10.0.3.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4099"
        },
        "Labels": null
    }
]

worker node:

[root@localhost ~]# docker network inspect  ha-network-overlay
[
    {
        "Name": "ha-network-overlay",
        "Id": "7x41wgxztm8zs7n4gslsfdd80",
        "Created": "2021-11-02T11:47:33.993144209+08:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.3.0/24",
                    "Gateway": "10.0.3.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "fd4d197d4e111a1caf64cbda5e09d82ecbd2858a57ccb0ca80fd36f7e26d7e56": {
                "Name": "test-ha-nginx",
                "EndpointID": "a8640e972a159086c2554e69368c0ce1eee4f9327d595c6820003057cb2bc29f",
                "MacAddress": "02:42:0a:00:03:02",
                "IPv4Address": "10.0.3.2/24",
                "IPv6Address": ""
            },
            "lb-ha-network-overlay": {
                "Name": "ha-network-overlay-endpoint",
                "EndpointID": "c3cecdd88456052244e522064c5f56c602043d74654c3a243db5b18f7569623d",
                "MacAddress": "02:42:0a:00:03:03",
                "IPv4Address": "10.0.3.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4099"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "f9c3fce6f5b9",
                "IP": "172.18.8.103"
            }
        ]
    }
]

网络的使用

version: '3.5'
services:
  ha-pxc01:
    restart: always
    image: percona/percona-xtradb-cluster:5.7.28-2
    container_name: ha-pxc01
    privileged: true
    ports:
    - 3306:3306
    environment:
        - MYSQL_ROOT_PASSWORD=123456
        - CLUSTER_NAME=ha-pxc
    volumes:
    - ../data/v1:/var/lib/mysql
    networks:
      - ha-network-overlay

networks:
   ha-network-overlay:
     external: true

部署架构

集群最擅长的就是解决多服务问题，只要在同一 network 之下，服务之间默认可以直接通过 service_name 互通有无。但为了避免混乱，各服务与外部的通信最好统一交给一个反向代理服务转发。

部署mysql 的PXC集群

第一个节点

version: '3.5'
services:
  ha-pxc01:
    restart: always
    image: percona/percona-xtradb-cluster:5.7.28-2
    container_name: ha-pxc01
    privileged: true
    ports:
    - 3306:3306
    environment:
        - MYSQL_ROOT_PASSWORD=123456
        - CLUSTER_NAME=ha-pxc
    volumes:
    - ../data/v1:/var/lib/mysql
    networks:
      - ha-network-overlay

networks:
   ha-network-overlay:
     external: true

第2/3个节点

version: '3.5'
services:
  ha-pxc02:
    restart: always
    image: percona/percona-xtradb-cluster:5.7.28-2
    container_name: ha-pxc02
    privileged: true
    ports:
    - 3307:3306
    environment:
    - MYSQL_ROOT_PASSWORD=123456
    - CLUSTER_NAME=ha-pxc
    - CLUSTER_JOIN=ha-pxc01
    volumes:
    - ../data/v2:/var/lib/mysql
    networks:
      - ha-network-overlay

networks:
   ha-network-overlay:
         external: true

导入镜像和创建目录

导入镜像

scp /root/percona-xtradb-cluster.tar  root@172.18.8.101:/root
scp /root/percona-xtradb-cluster.tar  root@172.18.8.103:/root
docker load -i /root/percona-xtradb-cluster.tar

创建目录


# 创建数据卷
mkdir -p /home/docker-compose/pxc-ha/data/{v1,v2,v3}/
chmod 777  /home/docker-compose/pxc-ha/data/{v1,v2,v3}/

mkdir -p /home/docker-compose/pxc-ha/pxc{01,02,03}
chmod 777  /home/docker-compose/pxc-ha/pxc{01,02,03}

复制文件，启动节点

第1个节点

[root@localhost pxc-ha]# sh start_node1.sh
===========stoping myql .....================
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
===========starting myql pxc01.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating ha-pxc01 ... done
=========== myql pxc01 starting.....================

第2个节点

[root@localhost pxc-ha]# sh start_node2.sh
===========stoping myql .....================
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
sed: can't read /home/docker-compose/pxc-ha/data/v1//grastate.dat: No such file or directory
===========starting myql pxc02.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating ha-pxc02 ... done
=========== myql pxc02 starting.....================

第三个节点

[root@k8s-master pxc-ha]# sh start_node3.sh
===========stoping myql .....================
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
Network ha-network-overlay is external, skipping
sed: can't read /home/docker-compose/pxc-ha/data/v1//grastate.dat: No such file or directory
===========starting myql pxc03.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating ha-pxc03 ... done
=========== myql pxc03 starting.....================

访问pxc节点和集群

第1个节点

172.18.8.101  3303

show status like 'wsrep_cluster%'

第2个节点

172.18.8.103  3303

show status like 'wsrep_cluster%'

第3个节点

172.18.8.104  3303

show status like 'wsrep_cluster%'

部署hapoxy

导入镜像和复制文件

导入镜像

scp /root/ha-nginx.tar  root@172.18.8.101:/root
scp /root/ha-nginx.tar  root@172.18.8.103:/root
scp /root/ha-nginx.tar  root@172.18.8.104:/root


docker load -i /root/ha-nginx.tar

scp /root/haproxy.tar  root@172.18.8.101:/root
scp /root/haproxy.tar  root@172.18.8.103:/root
docker load -i /root/haproxy.tar

复制 haproxy-ha的配置文件，和编排文件过去

编排文件

然后启动

启动之后测试

第1个节点的proxy

172.18.8.101  23303

show status like 'wsrep_cluster%'

部署高可用nacos集群

修改数据库的配置

初始化数据库 nacos_dev, 使用 init.sql脚本创建库

在nacos.env 环境变量文件中

TZ=Asia/Shanghai
JVM_XMS=256m
JVM_XMX=512m
JVM_XMN=128m
MODE=cluster
#PREFER_HOST_MODE=hostname
#NACOS_SERVERS=nacos1:8848 nacos2:8848 nacos3:8848
PREFER_HOST_MODE=ip
NACOS_AUTH_ENABLE=true
MYSQL_SERVICE_HOST=172.18.8.101
MYSQL_SERVICE_PORT=23303
MYSQL_SERVICE_DB_NAME=nacos_dev

MYSQL_SERVICE_USER=root
MYSQL_SERVICE_PASSWORD=123456
            

## pwd
NACOS_USER=nacos
NACOS_PASSWORD=nacos

## env
ORDER_ENV=dev
USER_ENV=dev

创建文件目录结构

批量创建目录的命令

# 创建数据卷


mkdir -p /home/docker-compose/nacos-ha/data_{01,02,03}/
chmod 777  /home/docker-compose/nacos-ha/data_{01,02,03}/


mkdir -p /home/docker-compose/nacos-ha/logs_{01,02,03}/
chmod 777  /home/docker-compose/nacos-ha/logs_{01,02,03}/

导入镜像

scp /root/nacos.tar  root@172.18.8.101:/root
scp /root/nacos.tar  root@172.18.8.103:/root
scp /root/nacos.tar  root@172.18.8.104:/root

docker load -i /root/nacos.tar

复制文件，启动节点

[root@localhost nacos-ha]# sh start_node1.sh
===========stoping nacos .....================
WARNING: Some services (nacos1, nacos2, nacos3) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Network ha-network-overlay is external, skipping
===========starting nacos node01================
WARNING: Some services (nacos1, nacos2, nacos3) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating nacos1 ... done
=========== starting nacos node01 .....================

清理一下内存

如果内存不够，清理一下内存

[root@localhost nacos-ha]# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        8.1G        925M        177M        6.5G        6.9G
Swap:          7.9G         22M        7.9G
[root@localhost nacos-ha]# echo 1 > /proc/sys/vm/drop_caches
[root@localhost nacos-ha]# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        8.0G        6.4G        177M        1.1G        7.0G
Swap:          7.9G         22M        7.9G

访问单个的nacos

http://172.18.8.101:8001/nacos/#/login
http://172.18.8.103:8002/nacos/#/login
http://172.18.8.104:8003/nacos/

用户名：nacos ，密码 nacos

增加代理的配置

global
  log 127.0.0.1 local0 info  # 设置日志文件输出定向
  daemon
  maxconn 20000
 
defaults
  log                     global
  option tcplog
  option dontlognull
  retries 5
  option redispatch
  mode tcp
  timeout queue 1m
  timeout connect 10s
  timeout client 1m  #客户端空闲超时时间
  timeout server 1m  #服务端空闲超时时间
  timeout check 5s
  maxconn 10000
 
#listen http_front #haproxy的客户页面
#  bind 0.0.0.0:1080
#  stats uri /haproxy?stats

listen stats
    mode http
    log global
    bind 0.0.0.0:1080
    stats enable
    stats refresh 30s
    stats uri /haproxy-stats
    # stats hide-version

listen nacos-cluster
   bind 0.0.0.0:8848
   mode tcp
   balance roundrobin
    server nacos_01 172.18.8.164:8001 check
    server nacos_02 172.18.8.164:8002 check
    server nacos_03 172.18.8.164:8003 check   
#   server nacos_01 nacos1:8848 check
#   server nacos_02 nacos2:8848 check
#   server nacos_03 nacos3:8848 check

在103上部署代理，然后启动

访问高可用代理的nacos

高可用代理

http://172.18.8.103:28848/nacos/#/configurationManagement?dataId=&group=&appName=&namespace=

用户名：nacos ，密码 nacos

部署高可用redis集群

修改编排文件

version: '3.5'
services:
 redis-ha-1:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6001/data:/data
  environment:
   - REDIS_PORT=6001

 redis-ha-2:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6002/data:/data
  environment:
   - REDIS_PORT=6002

 redis-ha-3:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6003/data:/data
  environment:
   - REDIS_PORT=6003

 redis-ha-4:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6004/data:/data
  environment:
   - REDIS_PORT=6004

 redis-ha-5:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6005/data:/data
  environment:
   - REDIS_PORT=6005

 redis-ha-6:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /home/docker-compose/redis-cluster-ha/conf/6006/data:/data
  environment:
   - REDIS_PORT=6006

创建文件目录结构

批量创建目录的命令



mkdir -p /home/docker-compose/redis-cluster-ha/conf/{7001,7002,7003,7004,7005,7006}/data

导入镜像

scp /root/redis-cluster.tar  root@172.18.8.101:/root
scp /root/redis-cluster.tar  root@172.18.8.103:/root
scp /root/redis-cluster.tar  root@172.18.8.104:/root

docker load -i /root/redis-cluster.tar 


scp /root/redis-trib.tar  root@172.18.8.101:/root
scp /root/redis-trib.tar  root@172.18.8.103:/root
scp /root/redis-trib.tar  root@172.18.8.104:/root

docker load -i /root/redis-trib.tar

复制文件，启动节点

启动虚拟机节点1

[root@localhost redis-cluster-ha]# sh start_node1.sh
===========stoping redis-cluster-ha .....================
===========starting redis 01 02.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current no

To deploy your application across the swarm, use `docker stack deploy`.

Creating redis-cluster-ha_redis-ha-1_1 ... done
Creating redis-cluster-ha_redis-ha-2_1 ... done
=========== starting  redis 01 02.....================

查看进程

[root@localhost redis-cluster-ha]# ps -ef | grep redis
polkitd    1142   1109  0 10:01 ?        00:00:00 redis-server *:6001 [cluster]
polkitd    1145   1110  0 10:01 ?        00:00:00 redis-server *:6002 [cluster]
root       1335 130010  0 10:02 pts/0    00:00:00 grep --color=auto redis
polkitd    2614   2547  0 Jun23 ?        10:43:10 redis-server *:6379
[

启动虚拟机节点2


[root@localhost redis-cluster-ha]# sh start_node2.sh
===========stoping redis-cluster-ha .....================
===========starting redis 03 04.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating redis-cluster-ha_redis-ha-3_1 ... done
Creating redis-cluster-ha_redis-ha-4_1 ... done
=========== starting  redis 03 04.....================
[root@localhost redis-cluster-ha]# ps -ef | grep redis
polkitd   22080  22062  0 Sep09 ?        06:22:27 redis-server *:6001 [cluster]
polkitd   22165  22148  0 Sep09 ?        06:48:27 redis-server *:6006 [cluster]
polkitd   22254  22238  0 Sep09 ?        06:38:52 redis-server *:6005 [cluster]
polkitd   22346  22329  0 Sep09 ?        06:19:08 redis-server *:6003 [cluster]
polkitd   22440  22423  0 Sep09 ?        06:20:51 redis-server *:6002 [cluster]
polkitd   22532  22515  0 Sep09 ?        06:47:53 redis-server *:6004 [cluster]
polkitd  130049 130017  1 10:20 ?        00:00:00 redis-server *:7003 [cluster]
polkitd  130053 130021  1 10:20 ?        00:00:00 redis-server *:7004 [cluster]
root     130124 124951  0 10:20 pts/0    00:00:00 grep --color=auto redis

查看进程如上

启动虚拟机节点3

[root@k8s-master ~]# cd /home/docker-compose/redis-cluster-ha/
[root@k8s-master redis-cluster-ha]# cd ..
[root@k8s-master docker-compose]# rm -rf redis-cluster-ha/
[root@k8s-master docker-compose]# mkdir -p /home/docker-compose/redis-cluster-ha/conf/{7001,7002,7003,7004,7005,7006}/data
[root@k8s-master docker-compose]#
[root@k8s-master docker-compose]# cd /home/docker-compose/redis-cluster-ha/
[root@k8s-master redis-cluster-ha]# sh start_node3.sh
===========stoping redis-cluster-ha .....================
===========starting redis 05 06.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating redis-cluster-ha_redis-ha-5_1 ... done
Creating redis-cluster-ha_redis-ha-6_1 ... done

查看进程

[root@k8s-master redis-cluster-ha]# ps -ef | grep redis
root      19366  19339  0 Sep13 ?        00:00:00 sudo -u redis redis-server /etc/redis.conf
polkitd   19696  19366  0 Sep13 ?        04:53:51 redis-server *:6379
polkitd   69926  69862  0 10:20 ?        00:00:00 redis-server *:7006 [cluster]
polkitd   69945  69878  0 10:20 ?        00:00:00 redis-server *:7005 [cluster]
root      74218 120247  0 10:21 pts/0    00:00:00 grep --color=auto redis

清理一下内存

如果内存不够，清理一下内存

[root@localhost nacos-ha]# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        8.1G        925M        177M        6.5G        6.9G
Swap:          7.9G         22M        7.9G
[root@localhost nacos-ha]# echo 1 > /proc/sys/vm/drop_caches
[root@localhost nacos-ha]# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        8.0G        6.4G        177M        1.1G        7.0G
Swap:          7.9G         22M        7.9G

使用redis-trib.rb创建redis 集群

上面只是启动了6个redis容器，并没有设置集群，通过下面的命令可以设置集群。

使用 redis-trib.rb create 命令完成节点握手和槽分配过程

docker run --rm -it inem0o/redis-trib create --replicas 1 hostip:6001 hostip:6002 hostip:6003 hostip:6004 hostip:6005 hostip:6006

#hostip  换成 主机的ip

docker run --rm -it inem0o/redis-trib create --replicas 1 172.18.8.101:7001 172.18.8.101:7002 172.18.8.103:7003 172.18.8.103:7004 172.18.8.104:7005 172.18.8.104:7006

–replicas 参数指定集群中每个主节点配备几个从节点，这里设置为1，

redis-trib.rb 会尽可能保证主从节点不分配在同一机器下，因此会重新排序节点列表顺序。

节点列表顺序用于确定主从角色，先主节点之后是从节点。

创建过程中首先会给出主从节点角色分配的计划，并且会生成报告

日志如下：

[root@localhost redis-cluster-ha]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.18.8.101:7001 172.18.8.101:7002 172.18.8.103:7003 172.18.8.103:7004 172.18.8.104:7005 172.18.8.104:7006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.18.8.101:7001
172.18.8.103:7003
172.18.8.104:7005
Adding replica 172.18.8.103:7004 to 172.18.8.101:7001
Adding replica 172.18.8.101:7002 to 172.18.8.103:7003
Adding replica 172.18.8.104:7006 to 172.18.8.104:7005
M: 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0 172.18.8.101:7001
   slots:0-5460 (5461 slots) master
S: 8c07f7dc329e2a527a3e2d660178386f398fc19f 172.18.8.101:7002
   replicates cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769
M: cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769 172.18.8.103:7003
   slots:5461-10922 (5462 slots) master
S: d1b0e02f58bef1546489a59a646bc32841605f1d 172.18.8.103:7004
   replicates 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0
M: 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815 172.18.8.104:7005
   slots:10923-16383 (5461 slots) master
S: b7877dfeda5a8bfeb0b4648a76284cff71a522e0 172.18.8.104:7006
   replicates 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815
Can I set the above configuration? (type 'yes' to accept):

注意：出现Can I set the above configuration? (type ‘yes’ to accept): 是要输入yes 不是Y

[root@localhost redis-cluster-ha]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.18.8.101:7001 172.18.8.101:7002 172.18.8.103:7003 172.18.8.103:7004 172.18.8.104:7005 172.18.8.104:7006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.18.8.101:7001
172.18.8.103:7003
172.18.8.104:7005
Adding replica 172.18.8.103:7004 to 172.18.8.101:7001
Adding replica 172.18.8.101:7002 to 172.18.8.103:7003
Adding replica 172.18.8.104:7006 to 172.18.8.104:7005
M: 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0 172.18.8.101:7001
   slots:0-5460 (5461 slots) master
S: 8c07f7dc329e2a527a3e2d660178386f398fc19f 172.18.8.101:7002
   replicates cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769
M: cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769 172.18.8.103:7003
   slots:5461-10922 (5462 slots) master
S: d1b0e02f58bef1546489a59a646bc32841605f1d 172.18.8.103:7004
   replicates 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0
M: 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815 172.18.8.104:7005
   slots:10923-16383 (5461 slots) master
S: b7877dfeda5a8bfeb0b4648a76284cff71a522e0 172.18.8.104:7006
   replicates 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 172.18.8.101:7001)
M: 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0 172.18.8.101:7001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: b7877dfeda5a8bfeb0b4648a76284cff71a522e0 172.18.8.104:7006@17006
   slots: (0 slots) slave
   replicates 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815
M: cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769 172.18.8.103:7003@17003
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: d1b0e02f58bef1546489a59a646bc32841605f1d 172.18.8.103:7004@17004
   slots: (0 slots) slave
   replicates 50be2ff38c47cd8e0d32ecfc701e77401a75f3d0
S: 8c07f7dc329e2a527a3e2d660178386f398fc19f 172.18.8.101:7002@17002
   slots: (0 slots) slave
   replicates cb5a8030b8dbc39056a3c7cf1e36b1fed06ee769
M: 7df8a1c9695c8099f4e3155b8cb0e9133e3cb815 172.18.8.104:7005@17005
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

访问单个的redis

http://172.18.8.101:7001/
http://172.18.8.103:7002
http://172.18.8.104:7005

来一波客户端连接操作，随便哪个节点，看看可否通过外部访问 Redis Cluster 集群。

至此使用多机环境基于 Docker Compose 搭建 Redis Cluster 就到这里。

Docker Compose 简化了集群的搭建，之前的方式就需要一个个去操作，而 Docker Compose 只需要一个 docker-compose up/down 命令的操作即可。

部署高可用rocketmq集群

部署架构

创建rocketmq文件目录结构

批量创建目录的命令

 
 rm -rf /home/docker-compose/rocketmq-ha
mkdir -p /home/docker-compose/rocketmq-ha/data/{broker-ha-a,broker-ha-b,namesrv-ha-a,namesrv-ha-b}/{logs,store}
chmod 777 /home/docker-compose/rocketmq-ha/data/{broker-ha-a,broker-ha-b,namesrv-ha-a,namesrv-ha-b}/{logs,store}

导入rocketmq镜像




scp /root/rocketmq.tar  root@172.18.8.101:/root
scp /root/rocketmq.tar  root@172.18.8.103:/root
scp /root/rocketmq.tar  root@172.18.8.104:/root



scp /root/rocketmq-console-ng.tar  root@172.18.8.101:/root
scp /root/rocketmq-console-ng.tar  root@172.18.8.103:/root
scp /root/rocketmq-console-ng.tar  root@172.18.8.104:/root

docker load -i /root/rocketmq.tar 
docker load -i /root/rocketmq-console-ng.tar

编排文件

version: '3.5'
services:
  rmqnamesrv-ha-a:
   image: apacherocketmq/rocketmq:4.6.0
   container_name: rmqnamesrv-ha-a
   ports: 
    - 19876:9876
   restart: always
   environment:
     TZ: Asia/Shanghai
     JAVA_OPTS: "-Duser.home=/opt"
     JAVA_OPT_EXT: "-server -Xms256m -Xmx1024m"      
   volumes:
    - /home/docker-compose/rocketmq-ha/data/namesrv-ha-a/logs:/opt/logs
    - /home/docker-compose/rocketmq-ha/data/namesrv-ha-a/store:/opt/store
   command: sh mqnamesrv
   networks:
     - ha-network-overlay
  rmqnamesrv-ha-b:
   image: apacherocketmq/rocketmq:4.6.0
   container_name: rmqnamesrv-ha-b
   restart: always
   ports: 
    - 19877:9876
   volumes:
    - /home/docker-compose/rocketmq-ha/data/namesrv-ha-b/logs:/opt/logs
    - /home/docker-compose/rocketmq-ha/data/namesrv-ha-b/store:/opt/store
   command: sh mqnamesrv
   environment:
     TZ: Asia/Shanghai
     JAVA_OPTS: "-Duser.home=/opt"
     JAVA_OPT_EXT: "-server -Xms256m -Xmx1024m"      
   networks:
     - ha-network-overlay
  rmqbroker-ha-a:
   image: apacherocketmq/rocketmq:4.6.0
   container_name: rmqbroker-ha-a
   restart: always
   ports:
    - 20911:20911
    - 20912:20912
   volumes:
    - /home/docker-compose/rocketmq-ha/data/broker-ha-a/logs:/opt/logs
    - /home/docker-compose/rocketmq-ha/data/broker-ha-a/store:/opt/store
    - /home/docker-compose/rocketmq-ha/conf/broker-ha-a.conf:/opt/rocketmq-4.6.0/conf/broker.conf
   environment:
    TZ: Asia/Shanghai
    NAMESRV_ADDR: "rmqnamesrv-ha-a:9876"
    JAVA_OPTS: "-Duser.home=/opt"
    JAVA_OPT_EXT: "-server -Xms256m -Xmx1024m" 
   command: sh mqbroker -c /opt/rocketmq-4.6.0/conf/broker.conf autoCreateTopicEnable=true &
   depends_on:
    - rmqnamesrv-ha-a
   networks:
     - ha-network-overlay
  rmqbroker-ha-b:
   image: apacherocketmq/rocketmq:4.6.0
   container_name: rmqbroker-ha-b
   restart: always
   ports:
    - 20921:20921
    - 20922:20922
   volumes:
    - /home/docker-compose/rocketmq-ha/data/broker-ha-b/logs:/opt/logs
    - /home/docker-compose/rocketmq-ha/data/broker-ha-b/store:/opt/store
    - /home/docker-compose/rocketmq-ha/conf/broker-ha-b.conf:/opt/rocketmq-4.6.0/conf/broker.conf
   environment:
    TZ: Asia/Shanghai
    NAMESRV_ADDR: "rmqnamesrv-ha-b:9876"
    JAVA_OPTS: "-Duser.home=/opt"
    JAVA_OPT_EXT: "-server -Xms256m -Xmx512m -Xmn128m"
   command: sh mqbroker -c /opt/rocketmq-4.6.0/conf/broker.conf autoCreateTopicEnable=true &
   depends_on:
    - rmqnamesrv-ha-b
   networks:
     - ha-network-overlay
  rmqconsole-ha:
   image: styletang/rocketmq-console-ng
   container_name: rmqconsole-ha
   restart: always
   ports:
    - 29001:9001
   environment:
     TZ: Asia/Shanghai
     JAVA_OPTS: "-Duser.home=/opt"
     JAVA_OPT_EXT: "-server -Xms256m -Xmx1024m"      
   environment:
    JAVA_OPTS: "-Drocketmq.namesrv.addr=172.18.8.103:19876;172.18.8.104:19877 -Dcom.rocketmq.sendMessageWithVIPChannel=false -Dserver.port=9001" 
   networks:
     - ha-network-overlay
networks:
   ha-network-overlay:
     external: true

配置文件

第1个节点

brokerClusterName = rocketmq-cluster
brokerName = broker-ha-a
brokerId = 0
#当前 broker 监听的 IP，broker所在宿主机的IP
brokerIP1 = 172.18.8.103
deleteWhen = 04
fileReservedTime = 48
brokerRole = ASYNC_MASTER
flushDiskType = ASYNC_FLUSH
#namesrvAddr的地址
namesrvAddr = 172.18.8.103:19876;172.18.8.104:19877
autoCreateTopicEnable = true
#接受客户端连接的监听端?，要跟后面docker-compose.yml中的ports配置匹配
listenPort = 20911

第2个节点

brokerClusterName = rocketmq-cluster
brokerName = broker-ha-b
brokerId = 0
#当前 broker 监听的 IP，broker所在宿主机的IP
brokerIP1 = 172.18.8.104
deleteWhen = 04
fileReservedTime = 48
brokerRole = ASYNC_MASTER
flushDiskType = ASYNC_FLUSH
#namesrvAddr的地址
namesrvAddr = 172.18.8.103:19876;172.18.8.104:19877
autoCreateTopicEnable = true
#接受客户端连接的监听端?，要跟后面docker-compose.yml中的ports配置匹配
listenPort = 20921

启动节点

脚本说明：

停止所有的服务
 sh stop_node.sh

启动第1个节点
sh ./start_node1.sh
启动第二个节点
sh start_node2.sh

查看日志

docker-compose logs -f

启动第1个节点

[root@localhost rocketmq-ha]# sh ./start_node1.sh
===========stoping rocketmq-ha .....================
Network ha-network-overlay is external, skipping
===========starting nameserver 01  brocker01  rmqconsole.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating rmqnamesrv-ha-a ... done
Creating rmqconsole-ha   ... done
Creating rmqbroker-ha-a  ... done
=========== starting  rocketmq in node 01================

启动第二个节点

[root@k8s-master rocketmq-ha]# sh start_node2.sh
===========stoping rocketmq-ha .....================
Network ha-network-overlay is external, skipping
===========starting nameserver 02  brocker02  rmqconsole.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating rmqconsole-ha   ... done
Creating rmqnamesrv-ha-b ... done
Creating rmqbroker-ha-b  ... done
=========== starting  rocketmq in node 02================

访问rockermq

访问rockermq 的控制台

http://172.18.8.103:29001/
http://172.18.8.104:29001/

部署高可用minio集群

修改minio编排文件

version: '3.5'

# starts 4 docker containers running minio server instances.
# using haproxy reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio1
    volumes:
      - ./data1-1:/data1
      - ./data1-2:/data2
    expose:
      - "9000"
      - "9991"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"   http://minio{1...6}:9991/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
  minio2:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio2
    volumes:
      - ./data2-1:/data1
      - ./data2-2:/data2
    expose:
      - "9000"
      - "9991"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"    http://minio{1...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
  minio3:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio3
    volumes:
      - ./data3-1:/data1
      - ./data3-2:/data2
    expose:
      - "9991"
      - "9000"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"   http://minio{1...6}:9991/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
  minio4:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio4
    volumes:
      - ./data4-1:/data1
      - ./data4-2:/data2
    expose:
      - "9000"
      - "9991"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"   http://minio{1...6}:9991/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
  minio5:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio5
    volumes:
      - ./data5-1:/data1
      - ./data5-2:/data2
    expose:
      - "9991"
      - "9000"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"   http://minio{1...6}:9991/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
  minio6:
    image: minio/minio:RELEASE.2020-05-16T01-33-21Z
    hostname: minio6
    volumes:
      - ./data6-1:/data1
      - ./data6-2:/data2
    expose:
      - "9000"
      - "9991"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
      MINIO_SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    command: server --address ":9991"   http://minio{1...6}:9991/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9991/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
     - ha-network-overlay
networks:
   ha-network-overlay:
     external: true

创建minio文件目录结构

批量创建目录的命令

mkdir -p /home/docker-compose/minio-ha/data{1,2,3,4,5,6}-{1,2}

导入镜像

scp /root/minio.tar  root@172.18.8.101:/root
scp /root/minio.tar  root@172.18.8.103:/root
scp /root/minio.tar  root@172.18.8.104:/root

docker load -i /root/minio.tar

启动节点

cd /home/docker-compose/minio-ha/
sh ./start_node1.sh
sh ./start_node2.sh
sh ./start_node3.sh
docker-compose logs -f

启动第一个节点

[root@localhost minio-ha]# sh ./start_node1.sh
===========stoping minio-ha .....================
Network ha-network-overlay is external, skipping
===========starting minio 01 02.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating minio-ha_minio1_1 ... done
Creating minio-ha_minio2_1 ... done
=========== starting  minio1   minio2.....================

启动第2个节点

[root@localhost rocketmq-ha]# cd /home/docker-compose/minio-ha/
[root@localhost minio-ha]# sh ./start_node2.sh
===========stoping minio-ha .....================
Network ha-network-overlay is external, skipping
===========starting minio 03 04.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating minio-ha_minio4_1 ... done
Creating minio-ha_minio3_1 ... done
=========== starting  minio 03 minio 04.....================

启动第3个节点

[root@localhost minio-ha]# sh ./start_node1.sh
===========stoping minio-ha .....================
Network ha-network-overlay is external, skipping
===========starting minio 05 06.....================
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating minio-ha_minio5_1 ... done
Creating minio-ha_minio6_1 ... done
=========== starting  minio1   minio2.....================

部署高可用ES集群

创建ES文件目录结构

批量创建目录的命令

mkdir -p  /home/docker-compose/elasticsearch7-ha/{coordinate1,coordinate2,coordinate3}-{logs,data}
mkdir -p  /home/docker-compose/elasticsearch7-ha/{master1,master2,master3}-{logs,data}
chmod -R 777 /home/docker-compose/elasticsearch7-ha

导入镜像

scp /root/elasticsearch-with-ik-icu.tar  root@172.18.8.101:/root
scp /root/elasticsearch-with-ik-icu.tar  root@172.18.8.103:/root
scp /root/elasticsearch-with-ik-icu.tar  root@172.18.8.104:/root



scp /root/kibana.tar  root@172.18.8.101:/root
scp /root/kibana.tar  root@172.18.8.103:/root
scp /root/kibana.tar  root@172.18.8.104:/root

docker load -i /root/kibana.tar
docker load -i /root/elasticsearch-with-ik-icu.tar 



docker load -i /root/rocketmq.tar 
docker load -i /root/rocketmq-console-ng.tar
docker load -i /root/minio.tar 
docker load   -i  /root/elasticsearch-with-ik-icu.tar
docker load   -i   /root/kibana.tar

修改编排文件

version: '3.5'
services:
  master-ha1:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: master-ha1
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./master1-data:/usr/share/elasticsearch/data:rw
     - ./config/master1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./master1-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29201:9200
      - 29301:9300
    networks:
     - ha-network-overlay
  master-ha2:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: master-ha2
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./master2-data:/usr/share/elasticsearch/data:rw
     - ./config/master2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./master2-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29202:9200
      - 29302:9300
    networks:
     - ha-network-overlay
  master-ha3:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: master-ha3
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./master3-data:/usr/share/elasticsearch/data:rw
     - ./config/master3.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./master3-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29203:9200
      - 29303:9300
    networks:
     - ha-network-overlay
  coordinate1:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: coordinate1
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./coordinate1-data:/usr/share/elasticsearch/data:rw
     - ./config/coordinate1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./coordinate1-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29204:9200
      - 29304:9300
    networks:
     - ha-network-overlay
  coordinate2:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: coordinate2
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./coordinate2-data:/usr/share/elasticsearch/data:rw
     - ./config/coordinate2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./coordinate2-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29205:9200
      - 29305:9300
    networks:
     - ha-network-overlay
  coordinate3:
    image: andylsr/elasticsearch-with-ik-icu:7.14.0
    container_name: coordinate3
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      nproc: 65535
      memlock: -1
    cap_add:
      - ALL
    privileged: true
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    volumes:
     - ./coordinate3-data:/usr/share/elasticsearch/data:rw
     - ./config/coordinate3.yml:/usr/share/elasticsearch/config/elasticsearch.yml
     - ./coordinate3-logs:/user/share/elasticsearch/logs:rw
    ports:
      - 29206:9200
      - 29306:9300
    networks:
     - ha-network-overlay
  kibana:
    image: docker.elastic.co/kibana/kibana:7.14.0
    container_name: kibana
    volumes:
      - ./config/kibana.yml:/usr/share/kibana/config/kibana.yml
    ports:
      - 15601:5601
    ulimits:
      nproc: 65535
      memlock: -1
    cap_add:
      - ALL
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    networks:
     - ha-network-overlay
networks:
   ha-network-overlay:
     external: true

修改系统配置

查看一下内存映射的上限配置

cat /etc/sysctl.conf

如果没有，则在/etc/sysctl.conf文件最后添加一行：

vi  /etc/sysctl.conf

vm.max_map_count=262144

立即生效, 执行：

/sbin/sysctl -p

启动节点

 cd /home/docker-compose/elasticsearch7-ha/

sh ./start_node1.sh
sh ./start_node2.sh
sh ./start_node3.sh
sh ./stop_node.sh
docker-compose logs -f

启动第一个节点

[root@localhost elasticsearch7-ha]# sh ./start_node1.sh
===========stoping elasticsearch7-ha .....================

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating coordinate1 ... done
Creating kibana      ... done
Creating master-ha1  ... done
=========== starting master-ha1   coordinate1  kibana .....================

启动第2个节点

[root@k8s-master minio-ha]#  cd /home/docker-compose/elasticsearch7-ha/
[root@k8s-master elasticsearch7-ha]# sh ./start_node2.sh


To deploy your application across the swarm, use `docker stack deploy`.

Creating coordinate3 ... done
Creating master-ha3  ... done
=========== starting master-ha2   coordinate2   .....================
[root@k8s-master elasticsearch7-ha]# docker-compose logs -f
WARNING: Some services (coordinate1, coordinate2, coordinate3, kibana, master-ha1, master-ha2, master-ha3) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Attaching to master-ha2, coordinate2

启动第3个节点

[root@k8s-master minio-ha]#  cd /home/docker-compose/elasticsearch7-ha/
[root@k8s-master elasticsearch7-ha]# sh ./start_node3.sh


To deploy your application across the swarm, use `docker stack deploy`.

Creating coordinate3 ... done
Creating master-ha3  ... done
=========== starting master-ha3   coordinate3   .....================
[root@k8s-master elasticsearch7-ha]# docker-compose logs -f
WARNING: Some services (coordinate1, coordinate2, coordinate3, kibana, master-ha1, master-ha2, master-ha3) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Attaching to master-ha3, coordinate3

访问es

访问kibana 的控制台

http://172.18.8.101:15601/

查看集群中的节点个数

# 查询API: 
GET _cat/nodes?v

响应信息如下:

#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.22.0.7           67          98   7    0.20    0.82     1.39 cdfhilmrstw -      master1
172.22.0.2           77          98   8    0.20    0.82     1.39 cdfhilmrstw *      master3
172.22.0.3           20          98   7    0.20    0.82     1.39 lr          -      coordinate1
172.22.0.4           68          98   7    0.20    0.82     1.39 cdfhilmrstw      master2
172.22.0.5           19          98   7    0.20    0.82     1.39 lr          -      coordinate2

增加pxc/es/minio的负载均衡代理配置

ha的配置文件如下

global
  log 127.0.0.1 local0 info  # 设置日志文件输出定向
  daemon
  maxconn 20000
 
defaults
  log                     global
  option tcplog
  option dontlognull
  retries 5
  option redispatch
  mode tcp
  timeout queue 1m
  timeout connect 10s
  timeout client 1m  #客户端空闲超时时间
  timeout server 1m  #服务端空闲超时时间
  timeout check 5s
  maxconn 10000
 
#listen http_front #haproxy的客户页面
#  bind 0.0.0.0:1080
#  stats uri /haproxy?stats

listen stats
    mode http
    log global
    bind 0.0.0.0:2080
    stats enable
    stats refresh 30s
    stats uri /haproxy-stats
    # stats hide-version

listen ha-pxc-cluster
    bind 0.0.0.0:23303
    mode tcp
    balance roundrobin
    server ha-pxc01 ha-pxc01:3306 check
    server ha-pxc02 ha-pxc02:3306 check
    server ha-pxc03 ha-pxc03:3306 check


listen ha-nacos-cluster
   bind 0.0.0.0:8848
   mode tcp
   balance roundrobin
   server nacos_01 nacos1:8001 check
   server nacos_02 nacos2:8002 check
   server nacos_03 nacos3:8003 check   


listen ha-minio-cluster
   bind 0.0.0.0:9991
   mode tcp
   balance roundrobin
   server minio1 minio1:9991 check
   server minio2 minio2:9991 check
   server minio3 minio3:9991 check
   server minio4 minio4:9991 check
   server minio5 minio5:9991 check
   server minio6 minio6:9991 check


listen ha-es-cluster
   bind 0.0.0.0:9200
   mode tcp
   balance roundrobin
   server coordinate1 coordinate1:9200 check
   server coordinate2 coordinate2:9200 check
   server coordinate3 coordinate2:9200 check

在103上部署代理，然后启动

启动节点

cd /home/docker-compose/haproxy-ha/

docker-compose up -d
docker-compose logs -f

访问minio的控制台

http://172.18.8.104:29991/minio/
http://172.18.8.103:29991/minio/

http://172.18.8.101:29991/minio/

用户名：AKIAIOSFODNN7EXAMPLE，密码：wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

访问高可用代理的nacos

高可用代理

http://172.18.8.104:28848/nacos/#/configurationManagement?dataId=&group=&appName=&namespace=

用户名：nacos ，密码 nacos

程序访问es 的client node

http://172.18.8.104:29200/

增加VIP保障高可用

Keepalived 是一款轻量级HA集群应用，它的设计初衷是为了做LVS集群的HA，即探测LVS健康情况，从而进行主备切换，不仅如此，还能够探测LVS代理的后端主机的健康状况，动态修改LVS转发规则。

当LVS进行主备切换的时候，对外提供服务的IP是如何做到切换的呢？

这就依赖于keepalived 所应用的vrrp协议，即Virtual Reduntant Routing Protocol，虚拟冗余路由协议。简单来讲，此协议是将IP设置在虚拟接口之上，根据一定的规则实现IP在物理主机上流动，即哪台主机可以占有该IP。

而且keepalived具有脚本调用接口，可通过脚本完成拓展功能。

部署架构

目的：进一步提高keepalived 可用性

设计：有3台keepalived 服务器 192.168.1.106 192.168.1.107 192.168.1.108。

vip 是192.168.1.249

效果：只要有一台keepalive 运行正常。vip 就可以访问

server	hostname	ip
keepalived	z6	192.168.1.106
keepalived	z7	192.168.1.107
keepalived	z8	192.168.1.108

非抢占模式默认是谁先开启vip 运行在哪台机器上的。如果发生故障转移，再比较priority

离线部署keepalived

首先安装keepalived，下载离线包的命令为

yum install --downloadonly --downloaddir=./ keepalived

已经提供了安装包，上传安装包到 linux，比如root目录

keepalived_rpm.tar.gz

解压缩，然后执行命令安装

cd /root/
tar -zxvf   keepalived_rpm.tar.gz
cd keepalived_rpm
rpm -Uvh --force --nodeps *rpm

输出

[root@localhost ~]# tar -zxvf   keepalived_rpm.tar.gz
keepalived_rpm/
keepalived_rpm/keepalived-1.3.5-19.el7.x86_64.rpm
keepalived_rpm/lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x86_64.rpm
keepalived_rpm/net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64.rpm
keepalived_rpm/net-snmp-libs-5.7.2-49.el7_9.1.x86_64.rpm

安装的输出

[root@localhost ~]# cd keepalived_rpm
[root@localhost keepalived_rpm]# rpm -Uvh --force --nodeps *rpm
warning: keepalived-1.3.5-19.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:net-snmp-libs-1:5.7.2-49.el7_9.1 ################################# [ 25%]
   2:lm_sensors-libs-3.4.0-8.20160601g################################# [ 50%]
   3:net-snmp-agent-libs-1:5.7.2-49.el################################# [ 75%]
   4:keepalived-1.3.5-19.el7          ################################# [100%]

编写haproxy的监控脚本

vi /etc/keepalived/check_haproxy.sh

#!/bin/bash
A=`ps -C haproxy --no-header |wc -l`
if [ $A -eq 0 ];then
systemctl stop keepalived
fi

上传脚本，测试一下脚本

sh /etc/keepalived/check_haproxy.sh

修改keepalived配置

cat /etc/keepalived/keepalived.conf

cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

复制准备好的文件到离线环境，或者直接编辑。

vi /etc/keepalived/keepalived.conf

global_defs {   #全局定义部分
   notification_email {  #设置警报邮箱
     acassen@firewall.loc     #邮箱1
     failover@firewall.loc    #邮箱2
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc    #设置发件人地址
   smtp_server 127.0.0.1   #设置smtp server地址
   smtp_connect_timeout 30   #设置smtp超时连接时间    以上参数可以不配置
   router_id HA_HAPROXY    #是Keepalived服务器的路由标识在一个局域网内，这个标识（router_id）是唯一的
} 

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
}

vrrp_instance VI_1 {  #VRRP实例定义区块名字是VI_1
    state BACKUP    #表示当前实例VI_1的角色状态这个状态只能有MASTER和BACKUP两种状态，并且需要大写这些字符, MASTER 为正式工作的状态，BACKUP为备用的状态
    interface ens33    #绑定ens33网卡
    virtual_router_id 101 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90           # 优先级，备服务器设置 90 
    advert_int 1           # 指定VRRP 心跳包通告间隔时间，默认1秒 ,MASTER与BACKUP之间通信检查的时间间隔，
    authentication {   #authentication为权限认证配置不要改动,同一vrrp实例的MASTER与BACKUP使用相同的密码才能正常通信。
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress {   #设置虚拟IP地址
  #      192.168.31.60/24
         172.18.8.231/24 
    } 
    track_script {
        check_haproxy
    } 
}

离线包里边有第一个节点的配置文件，复制到 /etc/keepalived/ 目录就行

cp /home/docker-compose/keepalive-ha/keepalived01.conf   /etc/keepalived/keepalived.conf

输出如下：

[root@localhost keepalive-ha]# pwd
/home/docker-compose/keepalive-ha
[root@localhost keepalive-ha]# cp /home/docker-compose/keepalive-ha/keepalived01.conf   /etc/keepalived/keepalived.conf
cp: overwrite ‘/etc/keepalived/keepalived.conf’? yes
[root@localhost keepalive-ha]# cat /etc/keepalived/keepalived.conf
global_defs {   #?????岿??
   notification_email {  #???t???????
     acassen@firewall.loc     #????1
     failover@firewall.loc    #????2
     sysadmin@firewall.loc
   }

重启 keepalive服务

命令如下



# systemctl enable keepalived

# systemctl start keepalived

# systemctl restart keepalived

# systemctl status  keepalived

输出如下：

[root@localhost ~]# systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@localhost ~]# systemctl restart keepalived
[root@localhost ~]#  systemctl status  keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-11-05 13:28:30 CST; 19s ago
  Process: 48837 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 48838 (keepalived)
    Tasks: 3
   Memory: 3.4M
   CGroup: /system.slice/keepalived.service
           ├─48838 /usr/sbin/keepalived -D
           ├─48839 /usr/sbin/keepalived -D
           └─48840 /usr/sbin/keepalived -D

Nov 05 13:28:34 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:34 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:34 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:34 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 ...1.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Nov 05 13:28:39 localhost.localdomain Keepalived_vrrp[48840]: Sending gratuitous ARP on ens33 for 192.168.31.60
Hint: Some lines were ellipsized, use -l to show in full.

查看IP

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:69:a4:84 brd ff:ff:ff:ff:ff:ff
    inet 172.18.8.101/24 brd 172.18.8.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 172.18.8.231/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b7be:346e:2f0f:859e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

访问高可用代理后的nacos

高可用代理

http://172.18.8.231:28848/nacos/#/configurationManagement?dataId=&group=&appName=&namespace=

用户名：nacos ，密码 nacos

离线部署keepalived到第2个节点，第3个节点

复制文件/并且安装

scp /root/keepalived_rpm.tar.gz  root@172.18.8.103:/root
scp /root/keepalived_rpm.tar.gz  root@172.18.8.104:/root

解压缩，然后执行命令安装

cd /root/
tar -zxvf   keepalived_rpm.tar.gz
cd keepalived_rpm
rpm -Uvh --force --nodeps *rpm

把离线包里的 keepalive-ha 传过去，然后复制到 /etc/keepalived/

第1个节点，执行

cp /home/docker-compose/keepalive-ha/keepalived01.conf   /etc/keepalived/keepalived.conf
cp /home/docker-compose/keepalive-ha/check_haproxy.sh   /etc/keepalived/

cat /etc/keepalived/keepalived.conf

第二个节点，执行

cp /home/docker-compose/keepalive-ha/keepalived02.conf   /etc/keepalived/keepalived.conf
cp /home/docker-compose/keepalive-ha/check_haproxy.sh   /etc/keepalived/


cat /etc/keepalived/keepalived.conf

输出如下：

[root@localhost keepalive-ha]# cp /home/docker-compose/keepalive-ha/keepalived02.conf   /etc/keepalived/keepalived.conf
cp: overwrite ‘/etc/keepalived/keepalived.conf’? yes
[root@localhost keepalive-ha]# cp /home/docker-compose/keepalive-ha/check_haproxy.sh   /etc/keepalived/
cp: overwrite ‘/etc/keepalived/check_haproxy.sh’? yes

第3个节点，执行

cp /home/docker-compose/keepalive-ha/keepalived03.conf   /etc/keepalived/keepalived.conf
cp /home/docker-compose/keepalive-ha/check_haproxy.sh   /etc/keepalived/


cat /etc/keepalived/keepalived.conf

输出如下：

cp /home/docker-compose/keepalive-ha/keepalived03.conf   /etc/keepalived/keepalived.conf
cp: overwrite ‘/etc/keepalived/keepalived.conf’? yes
[root@k8s-master keepalived_rpm]# cp /home/docker-compose/keepalive-ha/check_haproxy.sh   /etc/keepalived/
cp: overwrite ‘/etc/keepalived/check_haproxy.sh’? yes

重启 keepalive服务

命令如下


systemctl enable keepalived

systemctl start keepalived

systemctl restart keepalived

systemctl status  keepalived

systemctl stop  keepalived

查看IP地址

ip a

ip a | grep ens33

ifconfig ens33

输出结果


[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:69:a4:84 brd ff:ff:ff:ff:ff:ff
    inet 172.18.8.101/24 brd 172.18.8.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 172.18.8.231/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b7be:346e:2f0f:859e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

测试高可用

停止101上的haproxy

 sh ./stop_node.sh
 sh ./start_node.sh

[root@localhost haproxy-ha]# sh ./stop_node.sh
===========stoping haproxy-ha .....================
WARNING: Some services (ha_sit_proxy) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Stopping ha_sit_proxy ... done
Removing ha_sit_proxy ... done
Network ha-network-overlay is external, skipping