搭建lvs+keepalived集群遇到的问题

[李花怒放一树白]

我想使用lvs和keepalived来做httpd服务的负载均衡和HA,但是遇到了一些问题
先说环境
keepalived master
ip
192.168.3.211
hostname
hadoop-am
keepalived slave
ip
192.168.3.212
hostname
hadoop-sm
集群vip
192.168.3.210

lvs启动脚本1
vim /etc/init.d/LvsDR
#!/bin/bash
# Default-Start:3 5
# description: Start Cluster Scheduling
# create in 20070713 by dave
# In 20070718 fix the haresources call erro
# Source function library

. /etc/init.d/functions
prog="Cluster Scheduling"
#=====set the cluster ip===============
VIPT1=eth0:1
VIP1=192.168.3.210
RIP1=192.168.3.211
RIP2=192.168.3.212
#=====set functions====================
start() {
echo "Start the $prog"
ifconfig $VIPT1 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up && route add -host $VIP1 dev $VIPT1
ipvsadm -C
ipvsadm -A -t $VIP1:80 -s rr
ipvsadm -a -t $VIP1:80 -r $RIP1:80 -g
ipvsadm -a -t $VIP1:80 -r $RIP2:80 -g
success
ipvsadm -ln
}
stop() {
echo "Stop the $prog"
ipvsadm -C
ifconfig $VIPT1 down
success
}
restart() {
stop
start
}
state() {
ipvsadm -ln
}
reload() {
restart
}
#=====call function======================
case "$1" in
start)
start
;;
stop)
stop
;;
reload|restart)
restart
;;
status)
state
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac

lvs启动脚本2
vim /usr/local/edu/realserver
#!/bin/bash
# Default-Start: 3 5
# description: Start Cluster Scheduling
# create in 20070713 by dave
# Source function library
. /etc/init.d/functions
prog="Realserver"
#=====Set the Virtual IP=====
VIPT1=eth0:1
VIP1=192.168.3.210
#=====set functions=====
start() {
if [ ! -f /tmp/realserver.pid ]; then
echo "Start the $prog"
ifconfig $VIPT1 $VIP1 broadcast $VIP1 netmask 255.255.255.255 up
route add -host $VIP1 dev $VIPT1
touch /tmp/realserver.pid
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p
success
return 0
else echo "The $prog already running!!!"
return 1
failure
fi
}
stop() {
if [ ! -f /tmp/realserver.pid ]; then
echo "The $prog not run yeah!!!"
failure
return 1
else echo "Stop the $prog"
ifconfig $VIPT1 down
rm -rf /tmp/realserver.pid
success
return 0
fi
}
restart() {
stop
start
}
reload() {
restart
}
state() {
ifconfig $VIPT1
}
#=====call function======
case "$1" in
start)
restart
;;
stop)
stop
;;
reload|restart)
restart
;;
status)
state
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac

keepalived配置
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
[e-mail]
}
#notification_email_from [e-mail]
#smtp_server 192.168.200.1
#smtp_connect_timeout 30
#router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state MASTER/BACKUP
interface eth0
virtual_router_id 51
priority 100/80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.3.210
}
}

virtual_server 192.168.3.210 80 {
delay_loop 6
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP

real_server 192.168.3.211 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}

real_server 192.168.3.212 80 {
weight 2
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
在192.168.3.211上
# cat /var/www/html/index.html
Real Server 1
在192.168.3.212上
# cat /var/www/html/index.html
Real Server 2

============================分割线============================

遇到的问题
/etc/init.d/LvsDR、/usr/local/edu/realserver、/etc/init.d/keepalived这三个脚本启动都正常
而且httpd服务也可以正常启动
但是每次通过windows浏览器远程访问http://192.168.3.210总是只能访问到Real Server 1而不是Real Server 1和Real Server 2交替响应
或者等十几分钟后一定几率访问到Real Server 2
而且如果在两个节点的httpd都在运行的情况下，如果停止192.168.3.211上的httpd服务,则通过windows浏览器远程访问就会显示无法访问http://192.168.3.210而不是显示Real Server 2

以下是场景模拟的情况
[root@hadoop-am ~]# /etc/init.d/LvsDR start
Start the Cluster Scheduling
IP Virtual Server version 1.2.1 (size=4096) [ OK ]
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.3.210:80 rr
-> 192.168.3.211:80 Local 1 0 0
-> 192.168.3.212:80 Route 1 0 0
[root@hadoop-am ~]# /usr/local/edu/realserver start [ OK ]
Stop the Realserver
Start the Realserver [ OK ]
SIOCADDRT: File exists
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
[root@hadoop-sm ~]# /usr/local/edu/realserver start [ OK ]
Stop the Realserver
Start the Realserver [ OK ]
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
[root@hadoop-am ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@hadoop-sm ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@hadoop-am ~]# /etc/init.d/httpd start
Starting httpd: [ OK ]
[root@hadoop-sm ~]# /etc/init.d/httpd start
Starting httpd: [ OK ]
[root@hadoop-am ~]# echo '' > /var/log/messages
[root@hadoop-sm ~]# echo '' > /var/log/messages

停止hadoop-am上的httpd服务
[root@hadoop-am ~]# /etc/init.d/httpd stop
Stopping httpd: [ OK ]
[root@hadoop-sm ~]# tail -f /var/log/messages

Jun 13 05:05:05 hadoop-sm Keepalived_healthcheckers[5777]: TCP connection to [192.168.3.211]:80 failed !!!
Jun 13 05:05:05 hadoop-sm Keepalived_healthcheckers[5777]: Removing service [192.168.3.211]:80 from VS [192.168.3.210]:80
停止hadoop-sm上的httpd服务
[root@hadoop-sm ~]# /etc/init.d/httpd stop
Stopping httpd:
[root@hadoop-am ~]# tail -f /var/log/messages

Jun 14 20:51:25 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.211]:80 failed !!!
Jun 14 20:51:25 hadoop-am Keepalived_healthcheckers[6525]: Removing service [192.168.3.211]:80 from VS [192.168.3.210]:80
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.212]:80 failed !!!
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: Removing service [192.168.3.212]:80 from VS [192.168.3.210]:80
Jun 14 20:52:29 hadoop-am Keepalived_healthcheckers[6525]: Lost quorum 1-0=1 > 0 for VS [192.168.3.210]:80

启动hadoop-sm上的httpd服务
[root@hadoop-sm ~]# /etc/init.d/httpd start
Starting httpd: [ OK ]
[root@hadoop-am ~]# tail -f /var/log/messages

Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: TCP connection to [192.168.3.212]:80 success.
Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: Adding service [192.168.3.212]:80 to VS [192.168.3.210]:80
Jun 14 20:53:35 hadoop-am Keepalived_healthcheckers[6525]: Gained quorum 1+0=1 <= 1 for VS [192.168.3.210]:80

按理来说应该可以通过windows浏览器远程访问http://192.168.3.210并显示Real Server 2,但实际情况是windows浏览器远程访问显示http://192.168.3.210无法访问

问题究竟出在哪里，还请大神指教

[李花怒放一树白]

回2#防火墙早关了

[夏天__]

上网先查查 上网先查查

[anyonsss]

iptables 关闭了么。

[qq_16768371]

我也发生类似的问题了

[freedesert]

我跟你的情况一模一样 你是怎么解决的呢 可否告知一二？

[liuxu_8608]

从环境以及测试方式上看 vip和后端节点以及访问客户端应该是都处于一个网段的 另外就是 我看到keepalived的配置文件中使用的似乎是nat方式 这个方式 这种方式我没配过 不过 类似这个情况我之前配置dr模式的时候遇到过 当时情况基本也是如此 通过观察arp信息 发现 因为这些节点都在同一个广播域 而dr模式配置中 因为后端节点之间对客户端回包 需要更改数据包源地址为vip 因此 在lo网卡中也会绑定vip 在这种情况下 客户端发起arp请求时 会收到3台机器的回应 而一旦最终arp表对应的是后端某一个节点的mac 那么就不会再经过vip 而是之间去到那台后端服务器 只有在arp中的这个mac对应失效后才重新由vip分配后端 当时的解决方案就是 寻找或模拟通过路由器分割广播域后 再进行相关配置 可能写的有些乱 希望对你能够有些帮助