一步一步搭建Nginx 高可用
一、 准备工作
①、两台Centos7服务器
每台服务器分别配置两块网卡，一块内网、一块外网！内网ip分别为：10.20.11.80/10.20.11.81，外网IP作为VIP网卡。
Nginx_Master：10.20.11.80 eth192(内网网卡） eth224（外网网卡）
Nginx_Slave ：10.20.11.80 eth160(内网网卡） eth192（外网网卡）
②、内网IP需要取消默认网关。
2.1、10.20.11.80服务器
[root@nginx-174-master-80 keepalived]# nmcli connection modify ens192 ipv4.gateway ""
[root@nginx-174-master-80 keepalived]# nmcli connection down ens192 && nmcli connection up ens192
2.2、10.20.11.81服务器
[root@nginx-174-slave-81 keepalived]# nmcli connection modify ens160 ipv4.gateway ""
[root@nginx-174-slave-81 keepalived]# nmcli connection down ens160 && nmcli connection up ens160
Connection 'ens160' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/1)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)
③、分别安装好Nginx和keepalived
④、VIP 220.166.180.166 （测试IP）
二、 配置Nginx
①、安装并启动nginx(80、81两台服务器)
注：安装nginx省略
[root@nginx-174-master-80 vhost]# systemctl enable nginx
[root@nginx-174-master-80 vhost]# systemctl start nginx
[root@nginx-174-master-80 vhost]# systemctl status nginx
● nginx.service - nginx service
Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 15:49:07 CST; 4min 4s ago
Process: 14015 ExecStart=/usr/local/nginx/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 14016 (nginx)
CGroup: /system.slice/nginx.service
├─14016 nginx: master process /usr/local/nginx/sbin/nginx
├─14050 nginx: worker process
├─14051 nginx: worker process
├─14052 nginx: worker process
└─14053 nginx: worker process
Sep 06 15:49:07 nginx-174-master-80 systemd[1]: Starting nginx service...
Sep 06 15:49:07 nginx-174-master-80 nginx[14015]: nginx: [warn] 20480 worker_connections exceed open file resource limit: 10240
Sep 06 15:49:07 nginx-174-master-80 systemd[1]: Started nginx service.
②、配置vhost并复制到备份服务器
#### ③、防火墙放行Nginx端口（80、81两台服务器）
[root@nginx-174-master-80 vhost]# firewall-cmd --permanent --add-port=443/tcp
success
[root@nginx-174-master-80 vhost]# firewall-cmd --reload
success
[root@nginx-174-master-80 vhost]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: dhcpv6-client ssh
ports: 51022/tcp 80/tcp 443/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
三、 配置Keepalived
①、安装并设置开机启动（80、81两台服务器）
[root@nginx-174-master-80 vhost]# yum -y install keepalived
[root@nginx-174-master-80 vhost]# systemctl enable keepalived
②、10.20.11.80服务器配置keepalived
! Configuration File for keepalived
global_defs {
router_id sjyt_nginx174
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
inverval 3
}
vrrp_instance VI_1 {
state MASTER
interface ens192 #存活检测网卡，设置内网网卡
virtual_router_id 174
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass ***********
}
virtual_ipaddress {
220.166.180.166/26 dev ens224
}
track_script {
chk_nginx
}
virtual_routes {
default via 220.166.180.129
}
notify_master "/bin/python /etc/keepalived/nginx_keepalived_notify.py master"
notify_backup "/bin/python /etc/keepalived/nginx_keepalived_notify.py backup"
notify_fault "/bin/python /etc/keepalived/nginx_keepalived_notify.py fault"
}
③、10.20.11.81服务器配置keepalived
! Configuration File for keepalived
global_defs {
router_id sjyt_nginx174
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
inverval 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens160 #存活检测网卡，设置内网网卡
virtual_router_id 174
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass **************
}
virtual_ipaddress {
220.166.180.166/26 dev ens224
}
track_script {
chk_nginx
}
virtual_routes {
default via 220.166.180.129
}
notify_master "/bin/python /etc/keepalived/nginx_keepalived_notify.py master"
notify_backup "/bin/python /etc/keepalived/nginx_keepalived_notify.py backup"
notify_fault "/bin/python /etc/keepalived/nginx_keepalived_notify.py fault"
}
④、Nginx检测存活脚本
[root@nginx-174-master-80 keepalived]#vim /etc/keepalived/check_nginx.sh
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
n=`netstat -lntp | grep nginx | wc -l`
if [ $n -eq 0 ]; then
systemctl start nginx
n2=`netstat -lntp | grep nginx | wc -l`
if [ $n2 -eq 0 ]; then
echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
systemctl stop keepalived
fi
fi
⑤、Nginx 高可用切换钉钉通知
注：使用python脚本发送通知，两台机器都需要需要安装python requests 模块
5.1、安装epel-release
[root@localhost ~]# yum -y install epel-release
5.2、安装pip
[root@localhost ~]# yum -y install python-pip
5.3、安装requests
[root@localhost ~]# pip install requests
5.4、钉钉通知脚本
[root@nginx-174-master-80 keepalived]# cat nginx_keepalived_notify.py
#!/usr/local/python
# -*- coding: UTF-8 -*-
import requests
import sys
import json
import socket
import time
def info(jy):
# 钉钉告警
url = 'https://oapi.dingtalk.com/robot/send?access_token=****************'
headers = {
'Content-Type': 'application/json;charset=utf-8'
}
formdata = {
"msgtype": "text",
"text": {"content": str(jy)}
}
# print(formdata)
requests.post(url=url, data=json.dumps(formdata), headers=headers)
def change_status(status):
time1 = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time()))
hostname = socket.gethostname()
message = "Nginx高可用状态切换告警， " + str(time1) + " vrrp trasition , 主机: " + hostname + " Nginx 状态: change to be " + status
info(message)
if __name__ == '__main__':
status = sys.argv[1]
change_status(status)
⑥、防火墙放行keepalived
注：ens192为本机监听网口，需要根据每个服务器监听网口不通而改变。
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT;
firewall-cmd --reload;
⑦、启动keepalived
7.1、10.20.11.80服务器启动keepalived
[root@nginx-174-master-80 keepalived]# systemctl start keepalived
[root@nginx-174-master-80 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 16:31:16 CST; 3s ago
Process: 14951 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 14952 (keepalived)
CGroup: /system.slice/keepalived.service
├─14952 /usr/sbin/keepalived -D
├─14953 /usr/sbin/keepalived -D
└─14954 /usr/sbin/keepalived -D
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: VRRP_Instance(VI_1) setting protocol Virtual Routes
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:31:18 nginx-174-master-80 Keepalived_vrrp[14954]: Opening script file /bin/python
7.2 10.20.11.81服务器启动keepalived
[root@nginx-174-slave-81 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 16:47:50 CST; 20s ago
Process: 7677 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 7678 (keepalived)
CGroup: /system.slice/keepalived.service
├─7678 /usr/sbin/keepalived -D
├─7679 /usr/sbin/keepalived -D
└─7680 /usr/sbin/keepalived -D
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Instance(VI_1) removing protocol Virtual Routes
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: Using LinkWatch kernel netlink reflector...
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Instance(VI_1) removing protocol Virtual Routes
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: Opening script file /bin/python
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Script(chk_nginx) succeeded
Sep 06 16:47:50 nginx-174-slave-81 Keepalived_healthcheckers[7679]: Opening file '/etc/keepalived/keepalived.conf'.
注：如果发现服务器脑裂，就是两台服务器都处在MASTER状态，这种情况可能导致主、备切换不成功，需要考虑下是否防火墙阻挡了的原因。！
⑧、主Master（80）故障演练
8.1、停掉 80服务器nginx服务
[root@nginx-174-master-80 keepalived]# systemctl stop nginx
[root@nginx-174-master-80 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 16:45:06 CST; 5min ago
Process: 20884 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 20885 (keepalived)
CGroup: /system.slice/keepalived.service
├─20885 /usr/sbin/keepalived -D
├─20886 /usr/sbin/keepalived -D
└─20887 /usr/sbin/keepalived -D
Sep 06 16:45:08 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:08 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:08 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:08 nginx-174-master-80 Keepalived_vrrp[20887]: Opening script file /bin/python
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens224 for 220.166.180.166
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
Sep 06 16:45:13 nginx-174-master-80 Keepalived_vrrp[20887]: Sending gratuitous ARP on ens224 for 220.166.180.166
[root@nginx-174-master-80 keepalived]# systemctl status nginx
● nginx.service - nginx service
Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 16:50:54 CST; 18s ago
Process: 23381 ExecStop=/usr/local/nginx/sbin/nginx -s quit (code=exited, status=0/SUCCESS)
Process: 23394 ExecStart=/usr/local/nginx/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 23395 (nginx)
CGroup: /system.slice/nginx.service
├─23395 nginx: master process /usr/local/nginx/sbin/nginx
├─23396 nginx: worker process
├─23397 nginx: worker process
├─23398 nginx: worker process
└─23399 nginx: worker process
Sep 06 16:50:54 nginx-174-master-80 systemd[1]: Starting nginx service...
Sep 06 16:50:54 nginx-174-master-80 nginx[23394]: nginx: [warn] 20480 worker_connections exceed open file resource limit: 10240
Sep 06 16:50:54 nginx-174-master-80 systemd[1]: Started nginx service.
以上可以看出，当nginx挂掉之后keepalived会尝试拉起nginx！
8.2、我们直接测试关掉80服务器的网络。
主Master 挂掉前 Backup服务器的网络信息！
root@nginx-174-slave-81 keepalived]# ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.20.11.81 netmask 255.255.255.0 broadcast 10.20.11.255
inet6 fe80::c05f:7365:673:a5d3 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:9b:17:4b txqueuelen 1000 (Ethernet)
RX packets 43761 bytes 17186369 (16.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11836 bytes 1225993 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:50:56:9b:51:86 txqueuelen 1000 (Ethernet)
RX packets 12494 bytes 1183791 (1.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 768 bytes 63562 (62.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 32 bytes 2544 (2.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32 bytes 2544 (2.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@nginx-174-slave-81 keepalived]#
钉钉机器人已触发状态变更告警！
查看keepalived状态
[root@nginx-174-slave-81 keepalived]# ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.20.11.81 netmask 255.255.255.0 broadcast 10.20.11.255
inet6 fe80::c05f:7365:673:a5d3 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:9b:17:4b txqueuelen 1000 (Ethernet)
RX packets 44489 bytes 17238692 (16.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11943 bytes 1234091 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 220.166.180.166 netmask 255.255.255.192 broadcast 0.0.0.0
ether 00:50:56:9b:51:86 txqueuelen 1000 (Ethernet)
RX packets 12769 bytes 1211607 (1.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 889 bytes 74058 (72.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 32 bytes 2544 (2.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32 bytes 2544 (2.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@nginx-174-slave-81 keepalived]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-09-06 16:47:50 CST; 8min ago
Process: 7677 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 7678 (keepalived)
CGroup: /system.slice/keepalived.service
├─7678 /usr/sbin/keepalived -D
├─7679 /usr/sbin/keepalived -D
└─7680 /usr/sbin/keepalived -D
Sep 06 16:55:36 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:36 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:36 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:36 nginx-174-slave-81 Keepalived_vrrp[7680]: Opening script file /bin/python
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens192 for 220.166.180.166
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
Sep 06 16:55:41 nginx-174-slave-81 Keepalived_vrrp[7680]: Sending gratuitous ARP on ens192 for 220.166.180.166
以上，在主Nginx 网络故障后，备用nginx接管了公网IP。
8.2、恢复80服务器网络
钉钉通知状态已变更回来！
注：因为Nginx备用服务器没有设置网关，所以状态切换为Slave时触发钉钉通知失败！
