keepalived高可用集群
Keepalived 是一款基于 VRRP（Virtual Router Redundancy Protocol，虚拟路由冗余协议）实现的高可用解决方案，主要用于解决服务器单点故障问题，确保业务持续运行。它通过在多台服务器之间实现故障检测和自动切换，保障服务的高可用性。
keepalived部署
环境
关闭selinux和防火墙
KA1：172.25.254.50
[root@localhost ~]# grubby --update ALL kernel --args selinux=0[root@localhost ~]# reboot[root@localhost ~]# getenforceDisabled[root@localhost ~]# systemctl status firewalld○ firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled) Active: inactive (dead) Docs: man:firewalld(1)KA2：172.25.254.60
[root@localhost ~]# grubby --update ALL kernel --args selinux=0[root@localhost ~]# reboot[root@localhost ~]# getenforceDisabled[root@localhost ~]# systemctl status firewalld○ firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled) Active: inactive (dead) Docs: man:firewalld(1)安装keepalived
KA1：172.25.254.50
[root@localhost ~]# dnf intsall keepalived -y [root@localhost ~]# vim /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs { notification_email { wan@wan.org } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr # vrrp_strict vrrp_garp_interval 1 vrrp_gna_interval 1 vrrp_mcast_group4 224.0.0.44}vrrp_instance WEB_VIP { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }}[root@localhost ~]# systemctl enable --now keepalived.serviceCreated symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.[root@localhost ~]# systemctl restart keepalived.serviceKA2：172.25.254.60
[root@localhost ~]# dnf intsall keepalived -y [root@localhost ~]# vim /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs { notification_email { wan@wan.org } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr # vrrp_strict vrrp_garp_interval 1 vrrp_gna_interval 1 vrrp_mcast_group4 224.0.0.44}vrrp_instance WEB_VIP { state BACKUP interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }}[root@localhost ~]# systemctl enable --now keepalived.serviceCreated symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.bash[root@localhost ~]# systemctl restart keepalived.service查看VIP
KA1：172.25.254.50
KA2：172.25.254.60
抓包结果
启动keepalived日志
[root@localhost ~]# vim /etc/sysconfig/keepalivedKEEPALIVED_OPTIONS=\"-D -S 6\" #日志级别为0-7[root@localhost ~]# vim /etc/rsyslog.conflocal6.* /var/log/keepalived.log [root@localhost ~]# systemctl restart keepalived.service rsyslog.service [root@localhost ~]# tail -f /var/log/keepalived.log
实现独立子配置文件
当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中内容过多，不易管理 将不同集群的配置。
比如：不同集群的VIP配置放在独立的子配置文件中利用include 指令可以实现包含子配置文件
[root@localhost ~]# vim /etc/keepalived/keepalived.confinclude /etc/keepalived/conf.d/*.conf #添加内容[root@localhost ~]# mkdir /etc/keepalived/conf.d[root@localhost ~]# vim /etc/keepalived/conf.d/webvip.confvrrp_instance WEB_VIP { state BACKUP interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }}[root@localhost ~]# systemctl restart keepalived.servicekeepalived企业应用示例
非抢占模式
非抢占模式没有主，两个都是backup
KA1：172.25.254.50
[root@localhost ~]# vim /etc/keepalived/keepalived.confvrrp_instance WEB_VIP { state BACKUP #50也是backup interface ens160 virtual_router_id 51 priority 100 nopreempt #添加 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }}[root@localhost ~]# systemctl restart keepalived.serviceKA2：172.25.254.60
[root@localhost ~]# vim /etc/keepalived/keepalived.confvrrp_instance WEB_VIP { state BACKUP #60也是backup interface ens160 virtual_router_id 51 priority 80 nopreempt #添加 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }}[root@localhost ~]# systemctl restart keepalived.service
KA2也是
当KA1挂机时，KA2抢占到VIP
KA1：172.25.254.50
[root@localhost ~]# systemctl stop keepalived.serviceKA2：172.25.254.60
KA2挂机时，KA1抢占到VIP
抢占延迟模式
当172.25.254.50挂掉之后，172.25.254.60抢占到VIP，172.25.254.50恢复后，到设定的时间，VIP回到50
[root@localhost ~]# vim /etc/keepalived/keepalived.confstate MASTERpreempt_delay 10 # #指定抢占延迟时间为#s，默认延迟300s 5到10分钟[root@localhost ~]# systemctl restart keepalived.servicewatch -n 1 ipconfig 监控延迟出现的状态
VIP单播配置
KA1：172.25.254.50
[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { notification_email { wan@wan.org } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr # vrrp_strict vrrp_garp_interval 1 vrrp_gna_interval 1 #vrrp_mcast_group4 224.0.0.44 vrrp_ipsets keepalived #}vrrp_instance WEB_VIP { state MASTER # interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 } unicast_src_ip 172.25.254.50 # unicast_src_ip 172.25.254.50 # unicast_peer { # 172.25.254.60 # }}[root@localhost ~]# systemctl restart keepalived.service[root@localhost ~]# keepalived -f -t /etc/keepalived/keepalived.conf #查看配置是否有误KA2：172.25.254.60
[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { notification_email { wan@wan.org } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr #vrrp_strict vrrp_garp_interval 0 # vrrp_gna_interval 0 # #vrrp_mcast_group4 224.0.0.44 vrrp_ipsets keepalived #}vrrp_instance WEB_VIP { state BACKUPvrrp_instance WEB_VIP { state BACKUP interface ens160 virtual_router_id 51 priority 80 nopreempt advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 } unicast_src_ip 172.25.254.60 # unicast_peer { # 172.25.254.50 # }}[root@localhost ~]# systemctl restart keepalived.serviceKA1：172.25.254.50
172.25.254.60没有VIP
当172.25.254.50挂机时，172.25.254.60抢占到VIP
[root@localhost ~]# systemctl stop keepalived.serviceKA2：172.25.254.60
实现 Keepalived 状态切换的通知脚本
安装邮件发送工具
[root@localhost ~]# dnf install s-nail sendmail[root@localhost ~]# systemctl enable --now sendmail.serviceCreated symlink /etc/systemd/system/multi-user.target.wants/sendmail.service → /usr/lib/systemd/system/sendmail.service.Created symlink /etc/systemd/system/multi-user.target.wants/sm-client.service → /usr/lib/systemd/system/sm-client.service.网易云邮箱配置
[root@localhost ~]# vim /etc/mail.rcset smtp=smtp.163.comset smtp-auth=loginset smtp-auth-user=18082566241@163.comset smtp-auth-password=RGc4mi6GwAdhzzyfset from=18082566241@163.comset ssl-verify=ignore[root@localhost ~]# systemctl start sendmail#查看端口有没有被25占用[root@localhost ~]# netstat -antlupe | grep 25tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 111790 66794/sendmail: acctcp 0 64 172.25.254.50:22 172.25.254.1:60107 ESTABLISHED 0 28383 1825/sshd: root [prtcp 0 0 172.25.254.50:22 172.25.254.1:60108 ESTABLISHED 0 28398 1827/sshd: root [pr编辑状态转换脚本
[root@localhost ~]# mkdir -p /etc/keepalived/scripts[root@localhost ~]# vim -p /etc/keepalived/scripts/mail.sh#!/bin/bashmail_dest=\'594233887@qq.com\'mail_send(){ mail_subj=\"$HOSTNAME to be $1 vip 转移\" mail_mess=\"`date +%F\\ %T`: vrrp 转移,$HOSTNAME 变为 $1\" echo \"$mail_mess\" | mail -s \"$mail_subj\" $mail_dest}case $1 in master) mail_send master ;; backup) mail_send backup ;; fault) mail_send fault ;; *) exit 1 ;;esac[root@localhost ~]# chmod +x /etc/keepalived/scripts/mail.sh编辑keepalived配置文件
[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { notification_email { # wan@wan.org 18082566241@163.com } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr # vrrp_strict vrrp_garp_interval 1 vrrp_gna_interval 1 #vrrp_mcast_group4 224.0.0.44# vrrp_ipsets keepalived enable_scripts_security #添加 script_user root #添加}vrrp_instance WEB_VIP { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 } unicast_src_ip 172.25.254.50 unicast_peer { 172.25.254.60 } notify_master \"/etc/keepalived/mail.sh master\" #添加 notify_backup \"/etc/keepalived/mail.sh backup\" #添加 notify_fault \"/etc/keepalived/mail.sh fault\" #添加 }[root@localhost ~]# systemctl restart keepalived.service编辑主机文件修改主机名
[root@localhost ~]# vim /etc/hosts172.25.254.50 localhost.wan.org发送测试文件
[root@localhost ~]# echo hello | mailx -s text 18082566241@163.com邮箱收到信息
主主配置
KA1：172.25.254.50
[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { notification_email { wan@wan.org # 18082566241@163.com } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr # vrrp_strict vrrp_garp_interval 1 vrrp_gna_interval 1 vrrp_mcast_group4 224.0.0.44# vrrp_ipsets keepalived enable_scripts_security script_user root}vrrp_instance WEB_VIP { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }# unicast_src_ip 172.25.254.50# unicast_peer {# 172.25.254.60 }}vrrp_instance DB_VIP { state BACKUP interface ens160 virtual_router_id 52 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev ens160 label ens160:1 }# unicast_src_ip 172.25.254.50# unicast_peer {# 172.25.254.60 }}[root@localhost ~]# systemctl restart keepalived.service查看结果
KA2：172.25.254.60
[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { notification_email { wan@wan.org } notification_email_from wan@wan.org smtp_server 127.0.0.0 smtp_connect_timeout 30 router_id KA1 vrrp_skip_check_adv_addr #vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_mcast_group4 224.0.0.44 # vrrp_ipsets keepalived}vrrp_instance WEB_VIP { state BACKUP interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev ens160 label ens160:0 }# unicast_src_ip 172.25.254.60# unicast_peer {# 172.25.254.50# }}vrrp_instance DB_VIP { state MASTER interface ens160 virtual_router_id 52 priority 100 nopreempt advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev ens160 label ens160:1 }# unicast_src_ip 172.25.254.60# unicast_peer {# 172.25.254.50# }}[root@localhost ~]# systemctl restart keepalived.service查看结果
双主实现lvs高可用
RS1：172.25.254.10
[root@localhost ~]# sysctl -pnet.ipv4.ip_forward = 1net.ipv4.conf.all.arp_ignore = 1net.ipv4.conf.all.arp_announce = 2net.ipv4.conf.lo.arp_ignore = 1net.ipv4.conf.lo.arp_announce = 2[root@localhost ~]# ip a a 172.25.254.100/32 dev lo
RS2：172.25.254.11
[root@localhost ~]# sysctl -pnet.ipv4.ip_forward = 1net.ipv4.conf.all.arp_ignore = 1net.ipv4.conf.all.arp_announce = 2net.ipv4.conf.lo.arp_ignore = 1net.ipv4.conf.lo.arp_announce = 2[root@localhost ~]# ip a a 172.25.254.100/32 dev lo
KA1：172.25.254.50
[root@localhost ~]# vim /etc/keepalived/keepalived.confvirtual_server 172.25.254.100 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 172.25.254.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 2 retry 3 delay_before_retry 3 } } real_server 172.25.254.11 80 { weight 1 TCP_CHECK { connect_timeout 2 retry 3 delay_before_retry 3 connect_port 80 } }}[root@localhost ~]# systemctl restart keepalived.serviceKA2：172.25.254.60
[root@localhost ~]# vim /etc/keepalived/keepalived.confvirtual_server 172.25.254.100 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 172.25.254.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 2 retry 3 delay_before_retry 3 } } real_server 172.25.254.11 80 { weight 1 TCP_CHECK { connect_timeout 2 retry 3 delay_before_retry 3 connect_port 80 } }}[root@localhost ~]# systemctl restart keepalived.service
客户端测试
双主实现数据库的访问
RS1：172.25.254.10
RS2：172.25.254.11
KA1：172.25.254.50
KA2：172.25.254.60
客户端测试
vrrp实现主从角色替换
KA1：172.25.254.50
[root@localhost ~]# vim /mnt/text.sh#!/bin/bash[ ! -f \"/mnt/text\" ][root@localhost ~]# chmod +x /mnt/text.sh[root@localhost ~]# vim /etc/keepalived/keepalived.confglobal_defs { vrrp_script check_text { script \"/mnt/check_text.sh\" interval 1 weight -30 fall 2 rise 2 timeout 2 }} vrrp_instance WEB_VIP { track_script { #添加 check_text #添加 } #添加}[root@localhost ~]# systemctl restart keepalived.service测试
haproxy和haproxy实现主从替换
KA1：172.25.254.50
[root@localhost ~]# vim /etc/haproxy/haproxy.cfglisten webcluster bind *:80 mode http balance roundrobin server web1 172.25.254.10:80 check inter 3 fall 2 rise 3 server web2 172.25.254.11:80 check inter 3 fall 2 rise 3[root@localhost ~]# echo net.ipv4.ip_nonlocal_bind = 1 >>/etc/sysctl.conf[root@localhost ~]# sysctl -pnet.ipv4.ip_nonlocal_bind = 1[root@localhost ~]# systemctl enable --now haproxyCreated symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.KA2：172.25.254.60
[root@localhost ~]# vim /etc/haproxy/haproxy.cfglisten webcluster bind *:80 mode http balance roundrobin server web1 172.25.254.10:80 check inter 3 fall 2 rise 3 server web2 172.25.254.11:80 check inter 3 fall 2 rise 3[root@localhost ~]# echo net.ipv4.ip_nonlocal_bind = 1 >>/etc/sysctl.conf[root@localhost ~]# sysctl -pnet.ipv4.ip_nonlocal_bind = 1[root@localhost ~]# systemctl enable --now haproxyCreated symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.编写脚本
KA1：172.25.254.50
[root@localhost ~]# vim /etc/keepalived/scripts/haproxy.sh#!/bin/bash/usr/bin/killall -0 haproxy &> /dev/null[root@localhost ~]# chmod +x /etc/keepalived/scripts/haproxy.shKA2：172.25.254.60
[root@localhost ~]# vim /etc/keepalived/scripts/haproxy.sh#!/bin/bash/usr/bin/killall -0 haproxy &> /dev/null[root@localhost ~]# chmod +x /etc/keepalived/scripts/haproxy.sh测试
通告vrrp协议故障，说明keeplived挂机。优先级发生变化，说明haproxy挂机。
当172.25.254.50 haproxy 挂机之后，172.25.254.60抢夺到VIP
KA1：172.25.254.50
[root@localhost ~]# systemctl stop haproxy.serviceKA2：172.25.254.60
当开启之后，VIP回到172.25.254.50
[root@localhost ~]# systemctl start haproxy.serviceKA1：172.25.254.50
当172.25.254.60 haproxy 停止服务后，优先级降低，恢复服务后，优先级由50变为80
KA2：172.25.254.60
