一、集群简介
现状问题
业务发展过程中遇到的峰值瓶颈
- redis提供的服务OPS可以达到10万/秒，当前业务OPS已经达到10万/秒
- 内存单机容量达到256G，当前业务需求内存容量1T
使用集群的方式可以快速解决上述问题
集群架构
集群就是使用网络将若干台计算机联通起来，并提供统一的管理方式，使其对外呈现单机的服务效果
集群作用
- 分散单台服务器的访问压力，实现负载均衡
- 分散单台服务器的存储压力，实现可扩展性
- 降低单台服务器宕机带来的业务灾难
二、Redis集群结构设计
数据存储设计
- 通过算法设计，计算出key应该保存的位置
- 将所有的存储空间计划切割成16384份，每台主机保存一部分
每份代表的是一个存储空间，不是一个key的保存空间 - 将key按照计算出的结果放到对应的存储空间
问题：如果集群中有一个节点宕机或者集群中又加入一台新的节点呢？
- 增强可扩展性
集群内部通讯设计
- 各个数据库相互通信，保存各个库中槽的编号数据（槽用来区分数据的存储空间位置）
- 一次命中，直接返回
- 一次未命中，告知具体位置
三、cluster集群结构搭建
配置项说明：
cluster-enabled yes|no # 添加节点
cluster-config-file <filename> # cluster配置文件名，该文件属于自动生成，仅用于快速查找文件并查询文件内容
cluster-node-timeout <milliseconds> # 节点服务响应超时时间，用于判定该节点是否下线或切换为从节点
cluster-migration-barrier <count> # master连接的slave最小数量
节点操作命令：
cluster nodes # 查看集群节点信息
cluster replicate <master-id> # 进入一个从节点 redis，切换其主节点\
cluster meet ip:port # 发现一个新节点，新增主节点
cluster forget <id> # 忽略一个没有solt的节点
cluster failover # 手动故障转移
redis-trib命令:
redis-trib.rb add-node # 添加节点
redis-trib.rb del-node # 删除节点
redis-trib.rb reshard # 重新分片
配置集群为3主3从，使用一台机器的不同端口模拟3主3从进行
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6379.conf
port 6379
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6379.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6379.aof
cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6379.conf
38114:C 31 Aug 18:44:18.078 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38114:C 31 Aug 18:44:18.078 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38114, just started
38114:C 31 Aug 18:44:18.078 # Configuration loaded
38114:M 31 Aug 18:44:18.078 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38114:M 31 Aug 18:44:18.079 * No cluster configuration found, I'm 1a25e01038a516d023e3286180e2a5641e1e71a2
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 38114
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38114:M 31 Aug 18:44:18.081 # Server initialized
38114:M 31 Aug 18:44:18.081 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38114:M 31 Aug 18:44:18.081 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38114:M 31 Aug 18:44:18.081 * Ready to accept connections
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6380.conf
port 6380
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6380.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6380.aof
cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6380.conf
38118:C 31 Aug 18:44:19.890 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38118:C 31 Aug 18:44:19.890 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38118, just started
38118:C 31 Aug 18:44:19.890 # Configuration loaded
38118:M 31 Aug 18:44:19.890 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38118:M 31 Aug 18:44:19.892 * No cluster configuration found, I'm 4aa775e66e0e3e20862fe1f089468815267fa9a6
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6380
| `-._ `._ / _.-' | PID: 38118
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38118:M 31 Aug 18:44:19.893 # Server initialized
38118:M 31 Aug 18:44:19.893 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38118:M 31 Aug 18:44:19.893 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38118:M 31 Aug 18:44:19.893 * Ready to accept connections
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6381.conf
port 6381
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6381.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6381.aof
cluster-enabled yes
cluster-config-file nodes-6381.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6381.conf
38122:C 31 Aug 18:44:20.805 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38122:C 31 Aug 18:44:20.805 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38122, just started
38122:C 31 Aug 18:44:20.805 # Configuration loaded
38122:M 31 Aug 18:44:20.806 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38122:M 31 Aug 18:44:20.807 * No cluster configuration found, I'm dc9f6b27333c4390afda9d755ee0f9a0d9232a8c
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6381
| `-._ `._ / _.-' | PID: 38122
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38122:M 31 Aug 18:44:20.809 # Server initialized
38122:M 31 Aug 18:44:20.809 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38122:M 31 Aug 18:44:20.809 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38122:M 31 Aug 18:44:20.809 * Ready to accept connections
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6382.conf
port 6382
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6382.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6382.aof
cluster-enabled yes
cluster-config-file nodes-6382.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6382.conf
38126:C 31 Aug 18:44:21.689 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38126:C 31 Aug 18:44:21.689 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38126, just started
38126:C 31 Aug 18:44:21.689 # Configuration loaded
38126:M 31 Aug 18:44:21.689 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38126:M 31 Aug 18:44:21.690 * No cluster configuration found, I'm 4b714a226f90b050ba73e159d7c59b0bc8a55a14
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6382
| `-._ `._ / _.-' | PID: 38126
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38126:M 31 Aug 18:44:21.691 # Server initialized
38126:M 31 Aug 18:44:21.691 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38126:M 31 Aug 18:44:21.691 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38126:M 31 Aug 18:44:21.691 * Ready to accept connections
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6383.conf
port 6383
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6383.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6383.aof
cluster-enabled yes
cluster-config-file nodes-6383.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6383.conf
38130:C 31 Aug 18:44:22.555 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38130:C 31 Aug 18:44:22.555 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38130, just started
38130:C 31 Aug 18:44:22.555 # Configuration loaded
38130:M 31 Aug 18:44:22.555 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38130:M 31 Aug 18:44:22.557 * No cluster configuration found, I'm f81c67343afc7323523c9f76868c4022cff3d0b0
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6383
| `-._ `._ / _.-' | PID: 38130
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38130:M 31 Aug 18:44:22.558 # Server initialized
38130:M 31 Aug 18:44:22.558 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38130:M 31 Aug 18:44:22.558 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38130:M 31 Aug 18:44:22.558 * Ready to accept connections
[root@redis-master ~]# vim /usr/local/redis/redis-4.0.0/conf/redis-6384.conf
port 6384
daemonize no
dir /usr/local/redis/redis-4.0.0/data
dbfilename dump-6384.rdb
rdbcompression yes
rdbchecksum yes
appendfilename appendonly-6384.aof
cluster-enabled yes
cluster-config-file nodes-6384.conf
cluster-node-timeout 10000
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6384.conf
38134:C 31 Aug 18:44:24.793 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38134:C 31 Aug 18:44:24.793 # Redis version=4.0.0, bits=64, commit=00000000, modified=0, pid=38134, just started
38134:C 31 Aug 18:44:24.793 # Configuration loaded
38134:M 31 Aug 18:44:24.794 * Increased maximum number of open files to 10032 (it was originally set to 1024).
38134:M 31 Aug 18:44:24.794 * No cluster configuration found, I'm 570447233f1641d828d5dec61e91cb2dfcabcde6
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 4.0.0 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6384
| `-._ `._ / _.-' | PID: 38134
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
38134:M 31 Aug 18:44:24.796 # Server initialized
38134:M 31 Aug 18:44:24.796 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38134:M 31 Aug 18:44:24.796 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
38134:M 31 Aug 18:44:24.796 * Ready to accept connections
# 查看启动状态
[root@redis-master ~]# ps -ef | grep [r]edis
[root@redis-master ~]# ps -ef | grep [r]edis
root 38114 1007 0 18:44 pts/0 00:00:00 redis-server *:6379 [cluster]
root 38118 14710 0 18:44 pts/3 00:00:00 redis-server *:6380 [cluster]
root 38122 1045 0 18:44 pts/1 00:00:00 redis-server *:6381 [cluster]
root 38126 1064 0 18:44 pts/2 00:00:00 redis-server *:6382 [cluster]
root 38130 14905 0 18:44 pts/4 00:00:00 redis-server *:6383 [cluster]
root 38134 14943 0 18:44 pts/6 00:00:00 redis-server *:6384 [cluster]
安装ruby
ruby下载地址：http://www.ruby-lang.org/en/downloads/
[root@redis-master ~]# wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz
[root@redis-master ~]# mkdir /usr/local/ruby
[root@redis-master ~]# tar -xf ruby-2.3.1.tar.gz -C /usr/local/ruby
[root@redis-master ~]# cd /usr/local/ruby/ruby-2.3.1/
[root@redis-master ruby-2.3.1]# ./configure
[root@redis-master ruby-2.3.1]# make && make install
[root@redis-master ruby-2.3.1]# ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
[root@redis-master ruby-2.3.1]# gem -v
2.5.1
[root@redis-master ~]# wget http://www.zlib.net/zlib-1.2.11.tar.gz
[root@redis-master ~]# tar -zxf zlib-1.2.11.tar.gz -C /usr/local
[root@redis-master ~]# cd /usr/local/zlib-1.2.11/
[root@redis-master zlib-1.2.11]# ./configure
[root@redis-master zlib-1.2.11]# make
[root@redis-master zlib-1.2.11]# make install
[root@redis-master zlib]# cd /usr/local/ruby/ruby-2.4.6/ext/zlib
[root@redis-master zlib]# ruby ./extconf.rb --with-zlib-dir=/usr/local/zlib-1.2.11/
checking for deflateReset() in -lz... yes
checking for zlib.h... yes
checking for crc32_combine() in zlib.h... yes
checking for adler32_combine() in zlib.h... yes
checking for z_crc_t in zlib.h... yes
creating Makefile
[root@redis-master zlib]# cp Makefile Makefile-`date +%F`
[root@redis-master zlib]# vim Makefile
# 末行模式下输入 %s/$(top_srcdir)/..\/../g 进行替换，保存退出
[root@redis-master zlib]# make
compiling zlib.c
linking shared-object zlib.so
[root@redis-master zlib]# make install
/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux
[root@redis-master ~]# yum -y install openssl-devel
[root@redis-master ~]# cd /usr/local/ruby/ruby-2.4.6/ext/openssl/
[root@redis-master openssl]# ruby extconf.rb
[root@redis-master openssl]# cp Makefile Makefile-`date +%F`
[root@redis-master openssl]# vim Makefile
# 末行模式下输入 %s/$(top_srcdir)/..\/../g 进行替换，保存退出
[root@redis-master openssl]# make && make install
# 安装redis集群接口
[root@redis-master ~]# gem install redis
安装过程中有关错误解决方法：https://my.oschina.net/u/4075062/blog/3141609
创建集群
[root@redis-master ~]# /usr/local/redis/redis-4.0.0/src/redis-trib.rb create --replicas 1 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384
# --replicas 指定集群内部的结构
# 1 代表1个master连接1个slave；这里指定为1
# 2 代表1个master连接2个slave
# IP:PORT 指定redis服务节点及端口；因为这里指定的是1个master连接1个slave，且共有6个IP，那么写法为前3个为主后3个为从
>>> Creating cluster
/usr/local/lib/ruby/gems/2.4.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters: # 3个redis服务的master节点
127.0.0.1:6379
127.0.0.1:6380
127.0.0.1:6381
Adding replica 127.0.0.1:6382 to 127.0.0.1:6379 # 三个slave分别对应的master
Adding replica 127.0.0.1:6383 to 127.0.0.1:6380
Adding replica 127.0.0.1:6384 to 127.0.0.1:6381
M: 1a25e01038a516d023e3286180e2a5641e1e71a2 127.0.0.1:6379 # master节点对应的运行ID，IP，端口，分配的槽的数量和范围
slots:0-5460 (5461 slots) master
M: 4aa775e66e0e3e20862fe1f089468815267fa9a6 127.0.0.1:6380
slots:5461-10922 (5462 slots) master
M: dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 127.0.0.1:6381
slots:10923-16383 (5461 slots) master
S: 4b714a226f90b050ba73e159d7c59b0bc8a55a14 127.0.0.1:6382
replicates 1a25e01038a516d023e3286180e2a5641e1e71a2
S: f81c67343afc7323523c9f76868c4022cff3d0b0 127.0.0.1:6383
replicates 4aa775e66e0e3e20862fe1f089468815267fa9a6
S: 570447233f1641d828d5dec61e91cb2dfcabcde6 127.0.0.1:6384
replicates dc9f6b27333c4390afda9d755ee0f9a0d9232a8c
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 1a25e01038a516d023e3286180e2a5641e1e71a2 127.0.0.1:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: f81c67343afc7323523c9f76868c4022cff3d0b0 127.0.0.1:6383
slots: (0 slots) slave
replicates 4aa775e66e0e3e20862fe1f089468815267fa9a6
M: dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 127.0.0.1:6381
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 4b714a226f90b050ba73e159d7c59b0bc8a55a14 127.0.0.1:6382
slots: (0 slots) slave
replicates 1a25e01038a516d023e3286180e2a5641e1e71a2
M: 4aa775e66e0e3e20862fe1f089468815267fa9a6 127.0.0.1:6380
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 570447233f1641d828d5dec61e91cb2dfcabcde6 127.0.0.1:6384
slots: (0 slots) slave
replicates dc9f6b27333c4390afda9d755ee0f9a0d9232a8c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# 所有节点加入成功并且槽也分配完毕
# 查看节点配置文件内容（记录集群中各节点及槽的信息）
[root@redis-master ~]# cat /usr/local/redis/redis-4.0.0/data/nodes-6379.conf
1a25e01038a516d023e3286180e2a5641e1e71a2 127.0.0.1:6379@16379 myself,master - 0 1630407049000 1 connected 0-5460
f81c67343afc7323523c9f76868c4022cff3d0b0 127.0.0.1:6383@16383 slave 4aa775e66e0e3e20862fe1f089468815267fa9a6 0 1630407047000 5 connected
dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 127.0.0.1:6381@16381 master - 0 1630407048000 3 connected 10923-16383
4b714a226f90b050ba73e159d7c59b0bc8a55a14 127.0.0.1:6382@16382 slave 1a25e01038a516d023e3286180e2a5641e1e71a2 0 1630407049675 4 connected
4aa775e66e0e3e20862fe1f089468815267fa9a6 127.0.0.1:6380@16380 master - 0 1630407048658 2 connected 5461-10922
570447233f1641d828d5dec61e91cb2dfcabcde6 127.0.0.1:6384@16384 slave dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 0 1630407048000 6 connected
vars currentEpoch 6 lastVoteEpoch 0
# 其他节点配置文件基本类似
四、存取数据
[root@redis-master ~]# redis-cli
127.0.0.1:6379> set name zhangsan
(error) MOVED 5798 127.0.0.1:6380 # 错误，需要在对存的值计算出的槽所所对应的节点上进行操作
# 无法直接存数据，需要在连接时使用-c参数
[root@redis-master ~]# redis-cli -c
127.0.0.1:6379> set name zhangsan
-> Redirected to slot [5798] located at 127.0.0.1:6380 # 成功存入数据，但数据具体是在6380这台机器上存放
OK
# 同样再取数据时，如果要查找的数据存在指定节点时，可以直接使用redis-cli连接该节点进行查询
[root@redis-master ~]# redis-cli -p 6380
127.0.0.1:6380> get name
"zhangsan"
# 如果数据不在该节点直接使用reids-cli连接查询时，会报错
[root@redis-master ~]# redis-cli -p 6382
127.0.0.1:6382> get name
(error) MOVED 5798 127.0.0.1:6380
# 需要加 -c 参数
[root@redis-master ~]# redis-cli -p 6382 -c
127.0.0.1:6382> get name
-> Redirected to slot [5798] located at 127.0.0.1:6380
"zhangsan"
五、主从下线与主从切换
从节点掉线
# 停掉作为从的6382的redis服务，因为是前台启动，直接 ctrl + c 即可
^C38126:signal-handler (1630409852) Received SIGINT scheduling shutdown...
38126:S 31 Aug 19:37:32.365 # User requested shutdown...
38126:S 31 Aug 19:37:32.365 # Redis is now ready to exit, bye bye...
# 查看作为6382的主的6379输出信息
38114:M 31 Aug 19:37:32.366 # Connection with slave 127.0.0.1:6382 lost.
38114:M 31 Aug 19:37:43.915 * FAIL message received from 4aa775e66e0e3e20862fe1f089468815267fa9a6 about 4b714a226f90b050ba73e159d7c59b0bc8a55a14
# 与6382失去连接
# 其他4个节点信息均是一样的，如下
38118:M 31 Aug 19:37:43.914 * Marking node 4b714a226f90b050ba73e159d7c59b0bc8a55a14 as failing (quorum reached).
# 即，标记6382节点无法连接
# 此时再启动6382的redis服务
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6382.conf
……省略……
38181:S 31 Aug 19:42:11.382 # Cluster state changed: ok
38181:S 31 Aug 19:42:12.403 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:42:12.403 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:42:12.403 * Non blocking connect for SYNC fired the event.
38181:S 31 Aug 19:42:12.403 * Master replied to PING, replication can continue...
38181:S 31 Aug 19:42:12.403 * Trying a partial resynchronization (request 72e09746329ab3ccdae22b606630190513468146:1).
38181:S 31 Aug 19:42:12.403 * Successful partial resynchronization with master.
38181:S 31 Aug 19:42:12.403 * MASTER <-> SLAVE sync: Master accepted a Partial Resynchronization.
# 连接到主6379并同步信息成功
# 查看作为6382的主的6379输出信息
38114:M 31 Aug 19:42:11.419 * Clear FAIL state for node 4b714a226f90b050ba73e159d7c59b0bc8a55a14: slave is reachable again.
38114:M 31 Aug 19:42:12.403 * Slave 127.0.0.1:6382 asks for synchronization
38114:M 31 Aug 19:42:12.403 * Partial resynchronization request from 127.0.0.1:6382 accepted. Sending 3850 bytes of backlog starting from offset 1.
# 主重新连接上从
# 其他4个节点信息均是一样的，如下
38130:S 31 Aug 19:42:11.419 * Clear FAIL state for node 4b714a226f90b050ba73e159d7c59b0bc8a55a14: slave is reachable again.
# 标记6382节点上线
即：当 从节点下线后，其实对业务功能没有什么影响，其对应的主节点会对下线的从节点进行标记
主节点下线
# 停掉作为主的6379的redis服务，因为是前台启动，直接 ctrl + c 即可
^C38114:signal-handler (1630410651) Received SIGINT scheduling shutdown...
38114:M 31 Aug 19:50:51.306 # User requested shutdown...
38114:M 31 Aug 19:50:51.306 # Redis is now ready to exit, bye bye...
# 查看该主节点对应的从节点6382输出信息
38181:S 31 Aug 19:50:51.307 # Connection with master lost. # 丢失连接
38181:S 31 Aug 19:50:51.307 * Caching the disconnected master state. # 以下为一直尝试重新连接（具体重连次数、频率、时间由配置文件中的配置决定）
38181:S 31 Aug 19:50:52.217 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:52.217 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:52.217 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:53.244 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:53.244 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:53.244 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:54.265 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:54.265 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:54.266 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:55.286 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:55.286 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:55.286 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:56.312 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:56.312 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:56.312 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:57.334 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:57.334 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:57.334 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:58.355 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:58.355 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:58.355 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:50:59.380 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:50:59.381 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:50:59.381 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:51:00.396 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:51:00.396 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:51:00.396 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:51:01.414 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:51:01.414 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:51:01.414 # Error condition on socket for SYNC: Connection refused
38181:S 31 Aug 19:51:01.721 * FAIL message received from 4aa775e66e0e3e20862fe1f089468815267fa9a6 about 1a25e01038a516d023e3286180e2a5641e1e71a2
38181:S 31 Aug 19:51:01.721 # Cluster state changed: fail
38181:S 31 Aug 19:51:01.824 # Start of election delayed for 799 milliseconds (rank #0, offset 4550).
38181:S 31 Aug 19:51:02.437 * Connecting to MASTER 127.0.0.1:6379
38181:S 31 Aug 19:51:02.437 * MASTER <-> SLAVE sync started
38181:S 31 Aug 19:51:02.437 # Error condition on socket for SYNC: Connection refused # 在设定的时间内无法重新连接上，则认为6379主节点下线
38181:S 31 Aug 19:51:02.639 # Starting a failover election for epoch 7. # 该从节点就会将自己设为新的主
38181:S 31 Aug 19:51:02.641 # Failover election won: I'm the new master.
38181:S 31 Aug 19:51:02.641 # configEpoch set to 7 after successful failover
38181:M 31 Aug 19:51:02.641 # Setting secondary replication ID to 72e09746329ab3ccdae22b606630190513468146, valid up to offset: 1. New replication ID is e64419cbddf09d94da6ed4ac03a29f709c3f0109
38181:M 31 Aug 19:51:02.641 * Discarding previously cached master state.
38181:M 31 Aug 19:51:02.642 # Cluster state changed: ok
# 其他四个节点输出的信息均如下：
38118:M 31 Aug 19:51:01.721 * Marking node 1a25e01038a516d023e3286180e2a5641e1e71a2 as failing (quorum reached).
38118:M 31 Aug 19:51:01.721 # Cluster state changed: fail
38118:M 31 Aug 19:51:02.641 # Failover auth granted to 4b714a226f90b050ba73e159d7c59b0bc8a55a14 for epoch 7
38118:M 31 Aug 19:51:02.683 # Cluster state changed: ok
# 客户端中查看这两个节点对应的状态信息
[root@redis-master ~]# redis-cli -p 6382
127.0.0.1:6382> cluster nodes
dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 127.0.0.1:6381@16381 master - 0 1630411179000 3 connected 10923-16383
4aa775e66e0e3e20862fe1f089468815267fa9a6 127.0.0.1:6380@16380 master - 0 1630411181596 2 connected 5461-10922
4b714a226f90b050ba73e159d7c59b0bc8a55a14 127.0.0.1:6382@16382 myself,master - 0 1630411179000 7 connected 0-5460 # 可以看到，6382节点为主节点
1a25e01038a516d023e3286180e2a5641e1e71a2 127.0.0.1:6379@16379 master,fail - 1630410651408 1630410648029 1 disconnected # 因为无法连接，所以标记该节点为fail，但之前的角色没有改变
570447233f1641d828d5dec61e91cb2dfcabcde6 127.0.0.1:6384@16384 slave dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 0 1630411179548 6 connected
f81c67343afc7323523c9f76868c4022cff3d0b0 127.0.0.1:6383@16383 slave 4aa775e66e0e3e20862fe1f089468815267fa9a6 0 1630411180574 5 connected
# 6379节点上线
[root@redis-master ~]# redis-server /usr/local/redis/redis-4.0.0/conf/redis-6379.conf
……省略……
38203:M 31 Aug 20:04:42.863 # Configuration change detected. Reconfiguring myself as a replica of 4b714a226f90b050ba73e159d7c59b0bc8a55a14
38203:S 31 Aug 20:04:42.863 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
38203:S 31 Aug 20:04:42.863 # Cluster state changed: ok
38203:S 31 Aug 20:04:43.886 * Connecting to MASTER 127.0.0.1:6382
38203:S 31 Aug 20:04:43.886 * MASTER <-> SLAVE sync started
38203:S 31 Aug 20:04:43.886 * Non blocking connect for SYNC fired the event.
38203:S 31 Aug 20:04:43.887 * Master replied to PING, replication can continue...
38203:S 31 Aug 20:04:43.887 * Trying a partial resynchronization (request 72e09746329ab3ccdae22b606630190513468146:1).
38203:S 31 Aug 20:04:43.888 * Full resync from master: 992b55210ee590eec34c571d2f9615d72a133318:0
38203:S 31 Aug 20:04:43.888 * Discarding previously cached master state.
38203:S 31 Aug 20:04:43.985 * MASTER <-> SLAVE sync: receiving 175 bytes from master
38203:S 31 Aug 20:04:43.985 * MASTER <-> SLAVE sync: Flushing old data
38203:S 31 Aug 20:04:43.986 * MASTER <-> SLAVE sync: Loading DB in memory
38203:S 31 Aug 20:04:43.986 * MASTER <-> SLAVE sync: Finished with success
# 6382节点信息
38181:M 31 Aug 20:04:42.869 * Clear FAIL state for node 1a25e01038a516d023e3286180e2a5641e1e71a2: master without slots is reachable again.
38181:M 31 Aug 20:04:43.887 * Slave 127.0.0.1:6379 asks for synchronization
38181:M 31 Aug 20:04:43.887 * Unable to partial resync with slave 127.0.0.1:6379 for lack of backlog (Slave request was: 1).
38181:M 31 Aug 20:04:43.887 # Warning: slave 127.0.0.1:6379 tried to PSYNC with an offset that is greater than the master replication offset.
38181:M 31 Aug 20:04:43.887 * Starting BGSAVE for SYNC with target: disk
38181:M 31 Aug 20:04:43.887 * Background saving started by pid 38207
38207:C 31 Aug 20:04:43.889 * DB saved on disk
38207:C 31 Aug 20:04:43.889 * RDB: 0 MB of memory used by copy-on-write
38181:M 31 Aug 20:04:43.985 * Background saving terminated with success
38181:M 31 Aug 20:04:43.985 * Synchronization with slave 127.0.0.1:6379 succeeded
# 从以上信息可以看到，6379节点重新上线后是作为6382节点的从节点
# 客户端中再查看这两个节点对应的状态信息
[root@redis-master ~]# redis-cli -p 6382
127.0.0.1:6382> cluster nodes
dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 127.0.0.1:6381@16381 master - 0 1630411656000 3 connected 10923-16383
4aa775e66e0e3e20862fe1f089468815267fa9a6 127.0.0.1:6380@16380 master - 0 1630411655000 2 connected 5461-10922
4b714a226f90b050ba73e159d7c59b0bc8a55a14 127.0.0.1:6382@16382 myself,master - 0 1630411657000 7 connected 0-5460
1a25e01038a516d023e3286180e2a5641e1e71a2 127.0.0.1:6379@16379 slave 4b714a226f90b050ba73e159d7c59b0bc8a55a14 0 1630411656721 7 connected # 此时角色为slave，且未标记为fail
570447233f1641d828d5dec61e91cb2dfcabcde6 127.0.0.1:6384@16384 slave dc9f6b27333c4390afda9d755ee0f9a0d9232a8c 0 1630411657746 6 connected
f81c67343afc7323523c9f76868c4022cff3d0b0 127.0.0.1:6383@16383 slave 4aa775e66e0e3e20862fe1f089468815267fa9a6 0 1630411656000 5 connected
# 其他四个节点输出的信息均如下：
38134:S 31 Aug 20:04:42.869 * Clear FAIL state for node 1a25e01038a516d023e3286180e2a5641e1e71a2: master without slots is reachable again.
即：主节点下线后，作为该主节点的从节点会切换为主节点，当下线的主节点上线后，会作为切换后主节点的从节点