一、自定义 yaml 文件安装

背景

在 Kubernetes 中部署 Redis 集群面临挑战，因为每个 Redis 实例都依赖于一个配置文件，该文件可以跟踪其他集群实例及其角色。为此，需要结合使用 Kubernetes StatefulSets 和 PersistentVolumes 实现。Redis 集群架构如下图所示：

创建 StatefulSet yaml 文件

---
apiVersion:?v1
kind:?ConfigMap
metadata:
??name:?redis-cluster
data:
??update.sh:?|
????#!/bin/sh
????REDIS_NODES="/data/nodes.conf"
????sed?-i?-e?"/myself/?s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/"?${REDIS_NODES}
????exec?"$@"
??redis.conf:?|+
????bind?0.0.0.0
????cluster-enabled?yes
????cluster-require-full-coverage?no
????cluster-node-timeout?30000
????cluster-config-file?/data/nodes.conf
????cluster-migration-barrier?1
????appendonly?yes
????protected-mode?no
---
apiVersion:?apps.kruise.io/v1beta1
#?apiVersion:?apps/v1
kind:?StatefulSet
metadata:
??name:?redis-cluster
spec:
??serviceName:?redis-cluster
??replicas:?6
??selector:
????matchLabels:
??????app:?redis-cluster
??template:
????metadata:
??????labels:
????????app:?redis-cluster
????spec:
??????containers:
??????-?name:?redis
????????image:?redis:6.2.1-alpine
????????ports:
????????-?containerPort:?6379
??????????name:?client
????????-?containerPort:?16379
??????????name:?gossip
????????command:?["/conf/update.sh",?"redis-server",?"/conf/redis.conf"]
????????env:
????????-?name:?POD_IP
??????????valueFrom:
????????????fieldRef:
??????????????fieldPath:?status.podIP
????????volumeMounts:
????????-?name:?conf
??????????mountPath:?/conf
??????????readOnly:?false
????????-?name:?data
??????????mountPath:?/data
??????????readOnly:?false
??????volumes:
??????-?name:?conf
????????configMap:
??????????name:?redis-cluster
??????????defaultMode:?0755
??volumeClaimTemplates:
??-?metadata:
??????name:?data
????spec:
??????accessModes:?[?"ReadWriteOnce"?]
??????resources:
????????requests:
??????????storage:?100Gi
??????storageClassName:?rbd

解释：

cluster-migration-barrier?那些分配后仍然剩余 migration?barrier 个从节点的主节点才会触发节点分配，而不是分配前有 migration?barrier 个从节点的主节点就会触发节点分配，默认是 1，生产环境建议维持默认值。
protected-mode?no?参数是为了禁止外网访问 redis，如果启用了，则只能够通过 lookback?ip（127.0.0.1）访问 Redis，如果外网访问 redis，会报出异常。
apiVersion:?apps.kruise.io/v1beta1?控制器这里使用 kruise 提供的 Advanced?StatefulSet，如果集群没有安装 kruise，可以使用?apps/v1。

安装 redis 集群

[root@qd01-stop-k8s-master001?redis]#?kubectl?apply?-f?install-redis.yaml
configmap/redis-cluster?created
statefulset.apps.kruise.io/redis-cluster?created

[root@qd01-stop-k8s-master001?redis]#?kubectl?get?po?-n?op
NAME??????????????READY???STATUS????RESTARTS???AGE
redis-cluster-0???1/1?????Running???0??????????3m26s
redis-cluster-1???1/1?????Running???0??????????3m14s
redis-cluster-2???1/1?????Running???0??????????2m54s
redis-cluster-3???1/1?????Running???0??????????2m23s
redis-cluster-4???1/1?????Running???0??????????2m14s
redis-cluster-5???1/1?????Running???0??????????114s

创建 redis 集群的 service

---
apiVersion:?v1
kind:?Service
metadata:
??name:?redis-cluster
??namespace:?op
spec:
??type:?ClusterIP
??ports:
??-?port:?6379
????targetPort:?6379
????name:?client
??-?port:?16379
????targetPort:?16379
????name:?gossip
??selector:
????app:?redis-cluster

[root@qd01-stop-k8s-master001?redis]#?kubectl?apply?-f?redis-svc.yml
service/redis-cluster?created
[root@qd01-stop-k8s-master001?redis]#?kubectl?get?svc?-n?op
NAME????????????TYPE????????CLUSTER-IP??????EXTERNAL-IP???PORT(S)??????????????AGE
redis-cluster???ClusterIP???10.97.197.224???<none>????????6379/TCP,16379/TCP???9s

测试能后连通
[root@qd01-stop-k8s-master001?redis]#?telnet??10.97.197.224?6379
Trying?10.97.197.224...
Connected?to?10.97.197.224.
Escape?character?is?'^]'.

初始化 redis 集群

执行如下命令，获取到 pod IP，然后使用 redis-cli --cluster 创建集群

[root@qd01-stop-k8s-master001?redis]#?kubectl?-n?op?exec?-it?redis-cluster-0?--?redis-cli?--cluster?create?--cluster-replicas?1?$(kubectl?-n?op?get?pods?-l?app=redis-cluster?-o?jsonpath='{range.items[*]}{.status.podIP}:6379?{end}')
>>>?Performing?hash?slots?allocation?on?6?nodes...
Master[0]?->?Slots?0?-?5460
Master[1]?->?Slots?5461?-?10922
Master[2]?->?Slots?10923?-?16383
Adding?replica?100.88.43.67:6379?to?100.64.147.152:6379
Adding?replica?100.113.170.5:6379?to?100.98.174.217:6379
Adding?replica?100.64.147.153:6379?to?100.80.158.227:6379
M:?b47b27a3dbddf3fc1370cbe14ae753f4fce20b04?100.64.147.152:6379
???slots:[0-5460]?(5461?slots)?master
M:?09543217c903350e963fc4fdf4acb73f8a1b7f8b?100.98.174.217:6379
???slots:[5461-10922]?(5462?slots)?master
M:?5389ace495b68eeac85370d6783648dff68f2fb6?100.80.158.227:6379
???slots:[10923-16383]?(5461?slots)?master
S:?b1f39714c006ae55b12b18e6537303d7a00e1704?100.64.147.153:6379
???replicates?5389ace495b68eeac85370d6783648dff68f2fb6
S:?0113f4668ec2f3ca2e9470c44bd5faab532b0936?100.88.43.67:6379
???replicates?b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
S:?e1e2f18ae66c79f1943390beabb59613abbad38a?100.113.170.5:6379
???replicates?09543217c903350e963fc4fdf4acb73f8a1b7f8b
Can?I?set?the?above?configuration??(type?'yes'?to?accept):?yes
>>>?Nodes?configuration?updated
>>>?Assign?a?different?config?epoch?to?each?node
>>>?Sending?CLUSTER?MEET?messages?to?join?the?cluster
Waiting?for?the?cluster?to?join
..
>>>?Performing?Cluster?Check?(using?node?100.64.147.152:6379)
M:?b47b27a3dbddf3fc1370cbe14ae753f4fce20b04?100.64.147.152:6379
???slots:[0-5460]?(5461?slots)?master
???1?additional?replica(s)
S:?0113f4668ec2f3ca2e9470c44bd5faab532b0936?100.88.43.67:6379
???slots:?(0?slots)?slave
???replicates?b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
M:?09543217c903350e963fc4fdf4acb73f8a1b7f8b?100.98.174.217:6379
???slots:[5461-10922]?(5462?slots)?master
???1?additional?replica(s)
M:?5389ace495b68eeac85370d6783648dff68f2fb6?100.80.158.227:6379
???slots:[10923-16383]?(5461?slots)?master
???1?additional?replica(s)
S:?e1e2f18ae66c79f1943390beabb59613abbad38a?100.113.170.5:6379
???slots:?(0?slots)?slave
???replicates?09543217c903350e963fc4fdf4acb73f8a1b7f8b
S:?b1f39714c006ae55b12b18e6537303d7a00e1704?100.64.147.153:6379
???slots:?(0?slots)?slave
???replicates?5389ace495b68eeac85370d6783648dff68f2fb6
[OK]?All?nodes?agree?about?slots?configuration.
>>>?Check?for?open?slots...
>>>?Check?slots?coverage...
[OK]?All?16384?slots?covered.

验证 Redis 集群

[root@qd01-stop-k8s-master001?redis]#?kubectl?-n?op??exec?-it?redis-cluster-0?--?redis-cli?cluster?info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:178
cluster_stats_messages_pong_sent:181
cluster_stats_messages_sent:359
cluster_stats_messages_ping_received:176
cluster_stats_messages_pong_received:178
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:359

[root@qd01-stop-k8s-master001?redis]#?kubectl?-n?op??exec?-it?redis-cluster-0?--?redis-cli?cluster?nodes
0113f4668ec2f3ca2e9470c44bd5faab532b0936?100.88.43.67:6379@16379?slave?b47b27a3dbddf3fc1370cbe14ae753f4fce20b04?0?1615348311156?1?connected
09543217c903350e963fc4fdf4acb73f8a1b7f8b?100.98.174.217:6379@16379?master?-?0?1615348314162?2?connected?5461-10922
b47b27a3dbddf3fc1370cbe14ae753f4fce20b04?100.64.147.152:6379@16379?myself,master?-?0?1615348312000?1?connected?0-5460
5389ace495b68eeac85370d6783648dff68f2fb6?100.80.158.227:6379@16379?master?-?0?1615348312000?3?connected?10923-16383
e1e2f18ae66c79f1943390beabb59613abbad38a?100.113.170.5:6379@16379?slave?09543217c903350e963fc4fdf4acb73f8a1b7f8b?0?1615348313160?2?connected
b1f39714c006ae55b12b18e6537303d7a00e1704?100.64.147.153:6379@16379?slave?5389ace495b68eeac85370d6783648dff68f2fb6?0?1615348312158?3?connected

从输出可以看到，集群总共 6 个节点，三主三从

二、使用 kubeDB 安装

安装 KubeDB

获取 AppsCode License：https://license-issuer.appscode.com/；下载 KubeDB charts：https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001?kubedb-community]#?unzip?kubedb-community-v0.16.2.tgz
[root@qd01-stop-k8s-master001?kubedb-community]#?cd?kubedb-community
[root@qd01-stop-k8s-master001?kubedb-community]#?ls?-al
total?96
drwxr-xr-x?4?root?root???158?Mar?10?15:26?.
drwxr-xr-x?3?root?root????66?Mar?10?15:24?..
-rw-r--r--?1?root?root???351?Feb?16?09:55?Chart.yaml
drwxr-xr-x?2?root?root????28?Mar?10?15:24?ci
-rw-r--r--?1?root?root???493?Feb?16?09:55?doc.yaml
-rw-r--r--?1?root?root???353?Feb?16?09:55?.helmignore
-rw-r--r--?1?root?root?24422?Feb?16?09:55?README.md
drwxr-xr-x?2?root?root??4096?Mar?10?15:24?templates
-rw-r--r--?1?root?root?47437?Feb?16?09:55?values.openapiv3_schema.yaml
-rw-r--r--?1?root?root??5230?Feb?16?09:55?values.yaml

修改 values.yaml，把 License 文件放到 kubedb-community 目录下

使用 helm 安装

[root@qd01-stop-k8s-master001?kubedb-community]#?helm?install?kubedb-community?--namespace?kube-system?--set-file?license=./kubedb-community-license.txt?-f?values.yaml??.
NAME:?kubedb-community
LAST?DEPLOYED:?Wed?Mar?10?15:38:59?2021
NAMESPACE:?kube-system
STATUS:?deployed
REVISION:?1
TEST?SUITE:?None
NOTES:
To?verify?that?KubeDB?has?started,?run:
??kubectl?get?deployment?--namespace?kube-system?-l?"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
Now?install/upgrade?appscode/kubedb-catalog?chart.
To?install,?run:
??helm?install?kubedb-catalog?appscode/kubedb-catalog?--version?v0.16.2?--namespace?kube-system
To?upgrade,?run:
??helm?upgrade?kubedb-catalog?appscode/kubedb-catalog?--version?v0.16.2?--namespace?kube-system

执行如下命令查看是否安装完成
[root@qd01-stop-k8s-master001?kubedb-community]#?kubectl?get?deployment?--namespace?kube-system?-l?"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
NAME???????????????READY???UP-TO-DATE???AVAILABLE???AGE
kubedb-community???1/1?????1????????????1???????????38s

等待crds注册成功
[root@qd01-stop-k8s-master001?kubedb-community]#?kubectl?get?crds?-l?app.kubernetes.io/name=kubedb?-w
NAME???????????????????????????????????????CREATED?AT
elasticsearches.kubedb.com?????????????????2021-03-10T07:39:42Z
elasticsearchversions.catalog.kubedb.com???2021-03-10T07:39:45Z
etcds.kubedb.com???????????????????????????2021-03-10T07:39:42Z
etcdversions.catalog.kubedb.com????????????2021-03-10T07:39:45Z
memcacheds.kubedb.com??????????????????????2021-03-10T07:39:43Z
memcachedversions.catalog.kubedb.com???????2021-03-10T07:39:45Z
mongodbs.kubedb.com????????????????????????2021-03-10T07:39:43Z
mongodbversions.catalog.kubedb.com?????????2021-03-10T07:39:45Z
mysqls.kubedb.com??????????????????????????2021-03-10T07:39:43Z
mysqlversions.catalog.kubedb.com???????????2021-03-10T07:39:46Z
perconaxtradbs.kubedb.com??????????????????2021-03-10T07:39:43Z
perconaxtradbversions.catalog.kubedb.com???2021-03-10T07:39:46Z
pgbouncers.kubedb.com??????????????????????2021-03-10T07:39:44Z
pgbouncerversions.catalog.kubedb.com???????2021-03-10T07:39:46Z
postgreses.kubedb.com??????????????????????2021-03-10T07:39:44Z
postgresversions.catalog.kubedb.com????????2021-03-10T07:39:46Z
proxysqls.kubedb.com???????????????????????2021-03-10T07:39:44Z
proxysqlversions.catalog.kubedb.com????????2021-03-10T07:39:46Z
redises.kubedb.com?????????????????????????2021-03-10T07:39:45Z
redisversions.catalog.kubedb.com???????????2021-03-10T07:39:46Z

安装 KubeDB Catalog

先下载：https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001?kubedb-catalog]#?tar?-zxf?kubedb-catalog-v0.16.2.tgz
[root@qd01-stop-k8s-master001?kubedb-catalog]#?cd?kubedb-catalog
[root@qd01-stop-k8s-master001?kubedb-catalog]#?ls?-al
total?24
drwxr-xr-x??3?root?root??148?Mar?10?15:48?.
drwxr-xr-x??3?root?root???28?Mar?10?15:48?..
-rw-r--r--??1?root?root??321?Jan?26?20:08?Chart.yaml
-rw-r--r--??1?root?root??467?Jan?26?20:08?doc.yaml
-rw-r--r--??1?root?root??353?Jan?26?20:08?.helmignore
-rw-r--r--??1?root?root?3195?Jan?26?20:08?README.md
drwxr-xr-x?12?root?root??188?Mar?10?15:48?templates
-rw-r--r--??1?root?root??744?Jan?26?20:08?values.openapiv3_schema.yaml
-rw-r--r--??1?root?root?1070?Jan?26?20:08?values.yaml

[root@qd01-stop-k8s-master001?kubedb-catalog]#?helm?install?kubedb-catalog?--namespace?kube-system?-f?values.yaml??.
NAME:?kubedb-catalog
LAST?DEPLOYED:?Wed?Mar?10?15:50:50?2021
NAMESPACE:?kube-system
STATUS:?deployed
REVISION:?1
TEST?SUITE:?None

使用 kubedb 安装 Redis 集群

官方给的 Redis 生命周期如下图所示：

?kubedb 安装 redis 支持如下特性：

Features????Availability
Clustering?????
Instant?Backup?????
Scheduled?Backup?????
Persistent?Volume?????
Initialize?using?Snapshot?????
Initialize?using?Script?????
Custom?Configuration?????
Using?Custom?docker?image?????
Builtin?Prometheus?Discovery?????
Using?Prometheus?operator?????

查看支持的版本

[root@qd01-stop-k8s-master001?kubedb-catalog]#?kubectl?get?redisversions
NAME???????VERSION???DB_IMAGE????????????????DEPRECATED???AGE
4.0.11?????4.0.11????kubedb/redis:4.0.11??????????????????15m
4.0.6-v2???4.0.6?????kubedb/redis:4.0.6-v2????????????????15m
5.0.3-v1???5.0.3?????kubedb/redis:5.0.3-v1????????????????15m
6.0.6??????6.0.6?????kubedb/redis:6.0.6???????????????????15m

编辑 yaml 安装文件

可以参照：https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml。这里选择安装 6.0.6这个版本，我的集群 storageClassName: "rbd"，请根据实际修改。如果想自定义redis.conf，请参考：https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml。

apiVersion:?kubedb.com/v1alpha2
kind:?Redis
metadata:
??name:?redis-cluster
??namespace:?op
spec:
??version:?6.0.6
??mode:?Cluster
??cluster:
????master:?3
????replicas:?1
??storageType:?Durable
??storage:
????resources:
??????requests:
????????storage:?1Gi
????storageClassName:?"rbd"
????accessModes:
??????-?ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001?kubedb-community]#?kubectl?apply?-f?redis-cluster.yaml
redis.kubedb.com/redis-cluster?created

安装完成，可以如下查看
[root@qd01-stop-k8s-master001?kubedb-community]#?kubectl?get?rd,po?-n?op
NAME?????????????????????????????VERSION???STATUS?????????AGE
redis.kubedb.com/redis-cluster???6.0.6?????Provisioning???6m55s

NAME?????????????????????????READY???STATUS????RESTARTS???AGE
pod/redis-cluster-shard0-0???1/1?????Running???0??????????6m54s
pod/redis-cluster-shard0-1???1/1?????Running???0??????????6m18s
pod/redis-cluster-shard1-0???1/1?????Running???0??????????5m38s
pod/redis-cluster-shard1-1???1/1?????Running???0??????????5m1s
pod/redis-cluster-shard2-0???1/1?????Running???0??????????4m30s
pod/redis-cluster-shard2-1???1/1?????Running???0??????????4m8s

[root@qd01-stop-k8s-master001?redis]#?kubectl?get?svc?-n?op
NAME?????????????????TYPE????????CLUSTER-IP??????EXTERNAL-IP???PORT(S)????AGE
redis-cluster????????ClusterIP???10.97.197.224???<none>????????6379/TCP???5h16m
redis-cluster-pods???ClusterIP???None????????????<none>????????6379/TCP???17m

验证集群

[root@qd01-stop-k8s-master001?kubedb-community]#?kubectl?get?pods?-n??op??-o?jsonpath='{range.items[*]}{.metadata.name}?----------?{.status.podIP}:6379{"\t\n"}{end}'?|?grep?redis
redis-cluster-shard0-0?----------?100.64.147.156:6379
redis-cluster-shard0-1?----------?100.98.174.218:6379
redis-cluster-shard1-0?----------?100.126.252.204:6379
redis-cluster-shard1-1?----------?100.113.170.6:6379
redis-cluster-shard2-0?----------?100.107.55.69:6379
redis-cluster-shard2-1?----------?100.78.230.4:6379

[root@qd01-stop-k8s-master001?redis]#?kubectl?-n?op??exec?-it?redis-cluster-shard0-0??--?redis-cli?cluster?info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:864
cluster_stats_messages_pong_sent:882
cluster_stats_messages_sent:1746
cluster_stats_messages_ping_received:879
cluster_stats_messages_pong_received:864
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:1746

[root@qd01-stop-k8s-master001?redis]#?kubectl?-n?op??exec?-it?redis-cluster-shard0-0??--?redis-cli?cluster?nodes
1895cb4b9c31b848666c61000e502f55a29a8255?100.64.147.155:6379@16379?master?-?0?1615365162008?2?connected?5461-10922
30bdbf2ca37001774498a9b935afbc1cd2ce389c?100.126.252.203:6379@16379?slave?2c06092fafa99e0158e39e6237a04fed25be3550?0?1615365163000?1?connected
9b2cfbd5c1b417121d410141b6da9512ad29ce3c?100.78.230.3:6379@16379?slave?e83446c368839c5fdccf5f70e3b1004eb67cb651?0?1615365163512?3?connected
2c06092fafa99e0158e39e6237a04fed25be3550?100.82.197.130:6379@16379?myself,master?-?0?1615365162000?1?connected?0-5460
1379d2b20f26ab13d53068d276ec5d988b7a0273?100.64.122.197:6379@16379?slave?1895cb4b9c31b848666c61000e502f55a29a8255?0?1615365163000?2?connected
e83446c368839c5fdccf5f70e3b1004eb67cb651?100.107.55.68:6379@16379?master?-?0?1615365164014?3?connected?10923-16383