了解知识
基于上面的MYSQL集群,可以达到MYSQL数据库完整的备份及同时可读可写,现在要实现高可用,使用lvs heartbeat ldirectord mysql集群。
作用介绍:
Cluster :实现分布式计算环境的高实用、高冗余
LVS :对web服务器按照一定的规则进行转发
Heartbeat:提供ip漂移服务,在主服务器宕机后,备份服务器接管虚拟ip
Ldirectord:提供lvs控制,监控真实服务器,出现问题后,对轮循列表进行更新,去除出现问题的服务器
达到的预期效果:
1、实现mysql集群数据库完整的备份
2、实现mysql集群同时可读可写
3、监控NDB及SQL节点
4、使用同一个虚拟IP访问mysql集群,即LVS-DR-VIP
5、实现Mysql集群的负载均衡
6、实现LVS-DR-Master 备份功能的 LVS-DR-BACKUP
7、实时监控LVS-DR-Master
8、当LVS-DR-Master 出现故障时, LVS-DR-BACKUP及时接管
9、实时监控MYSQL节点,当出现故障时及时删除转发规则
一、IP信息列表:
1、需求IP信息列表
名称 IP
LVS-DR-VIP 10.0.0.100
LVS-DR-Master 10.0.0.133
LVS-DR-BACKUP 10.0.0.162
MySQL1-Realserver 10.0.0.151
MySQL2-Realserver 10.0.0.152
GateWay 10.0.0.1
其他(Mysql集群cluster,部署LVS过程中无需配置):
MGM管理节点 10.0.0.168
NDBD存储节点1 10.0.0.169
NDBD存储节点2 10.0.0.167
2、实验环境复用节点
MGM LVS-DR-Master
NDBD存储节点1 MYSQLD MySQL1-Realserver
NDBD存储节点2 MYSQLD MySQL2-Realserver
3、实验IP信息列表
hostname IP地址 应用服务
LVS-DR-VIP 10.0.0.100 LVS-DR-VIP
LVS-DR-Master 10.0.0.133 LVS-DR-Master、MGM
LVS-DR-BACKUP 10.0.0.162 LVS-DR-BACKUP、MYSQLD
MySQL1-Realserver 10.0.0.151 NDBD1、MYSQLD
MySQL2-Realserver 10.0.0.152 NDBD2、MYSQLD
GateWay 10.0.0.1
二、系统设置
OS:RHEL 5.4
预装软件:默认安装
1、配置两台DR服务器(LVS-DR-Master,LVS-DR-BACKUP)/etc/hosts文件
/etc/hosts 内容如下:
127.0.0.1 localhost.localdomain localhost
10.0.0.100 LVS-DR-VIP
10.0.0.133 LVS-DR-Master
10.0.0.162 LVS-DR-BACKUP
2、关掉一些不必要的系统服务
 /sbin/chkconfig acpid off
/sbin/chkconfig anacron off
/sbin/chkconfig apmd off
/sbin/chkconfig auditd off
/sbin/chkconfig autofs off
/sbin/chkconfig bluetooth off
/sbin/chkconfig cpuspeed off
/sbin/chkconfig cups off
/sbin/chkconfig gpm off
/sbin/chkconfig haldaemon off
/sbin/chkconfig iptables off
/sbin/chkconfig isdn off
/sbin/chkconfig kudzu off
/sbin/chkconfig mDNSResponder off
/sbin/chkconfig mdmonitor off
/sbin/chkconfig messagebus off
/sbin/chkconfig netfs off
/sbin/chkconfig nfslock off
/sbin/chkconfig nifd off
/sbin/chkconfig pcmcia off
/sbin/chkconfig portmap off
/sbin/chkconfig rhnsd off
/sbin/chkconfig rpcgssd off
/sbin/chkconfig rpcidmapd off
/sbin/chkconfig sendmail off
/sbin/chkconfig xfs off
在LVS-DR-BACKUP上同样设置
3、同步系统时间
#ntpdate time.nist.gov
在LVS-DR-BACKUP上同样设置
三、软件安装
1、更新源
scp /etc/yum.repos.dha.cf /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.*/authkeys /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.*/haresources /etc/ha.d/
cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /haresources /etc/ha.d/
四、配置LVS-DR-Master
1、配置ha.cf
/etc/ha.d/ha.cf 文件内容如下
 
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
ucast eth0 10.0.0.162 //备份机这个是主服务器的地址即 ucast eth0 10.0.0.133
auto_failback on
watchdog /dev/watchdog
node LVS-DR-Master
node LVS-DR-BACKUP
ping 10.0.0.100
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
2、配置authkeys
/etc/ha.d/authkeys 内容如下
 
#
# Authentication file. Must be mode 600
#
#
# Must have exactly one auth directive at the front.
# auth send authentication using this method-id
#
# Then, list the method and key that go with that method-id
#
# Available methods: crc sha1, md5. Crc doesn't need/want a key.
#
# You normally only have one authentication method-id listed in this file
#
# Put more than one to make a smooth transition when changing auth
# methods and/or keys.
#
#
# sha1 is believed to be the "best", md5 next best.
#
# crc adds no security, except from packet corruption.
# Use only on physically secure networks.
#
auth 1
1 crc
#2 sha1 HI!
#3 md5 Hello!
3、修改/etc/ha.d/authkeys权限改为600
chmod 600 /etc/ha.d/authkeys
4,配置haresources
修改/etc/ha.d/haresources 文件 ,在最后一行添加内容
LVS-DR-Master IPaddr::10.0.0.100/24/eth0:0 ldirectord
至此,heartbeat 配置完成,以下配置ldirectord
5、配置ldirectord
/etc/ha.d/ldirectord.cf 内容如下
checktimeout=3
checkinterval=1
#fallback=127.0.0.1:80
autoreload=yes
logfile="/var/log/ldirectord.log"
logfile="local0"
emailalert="liuguan269@163.com"
emailalertfreq=3600
emailalertstatus=all
quiescent=yes
#Sample configuration for a MySQL virtual service.
virtual = 10.0.0.100:3306
# real=master.up.com->slave.up.com:3306 gate 10
real=10.0.0.151:3306 gate 10
real=10.0.0.152:3306 gate 10
# fallback=127.0.0.1:3306
service=mysql
scheduler=wrr
# #persistent=600
# #netmask=255.255.255.255
protocol=tcp
checktype=negotiate
login="client"
passwd="password123"
database="cluster"
request="SELECT i FROM test"
receive="233" //可要可不要
6、打开ip_forward转发
修改/etc/sysctl.conf文件中的
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
在shell环境下执行
sysctl -p
7、脚本switchdr
#!/bin/bash
# description: start LVS of DirectorServer
#Written by :NetSeek http://www.linuxtone.org
GW=10.0.0.1 #当前局域网网关ip
# website director vip.
MySQL_VIP=10.0.0.100 #MySQL虚拟IP地址
MySQL_RIP1=10.0.0.151
MySQL_RIP2=10.0.0.152
/etc/rc.d/init.d/functions
logger $0 called with $1 #记录到系统日志messages
case "$1" in
start)
# set squid vip
/sbin/ipvsadm --set 30 5 60 #分别为协议tcp tcpfin udp 的超时时间
/sbin/ifconfig eth0:0 $MySQL_VIP broadcast $MySQL_VIP netmask 255.255.255.255 broadcast $MySQL_VIP up #新建VIP
#/sbin/ifconfig eth0:0 10.0.0.100 broadcast 10.0.0.100 netmask 255.255.255.255 broadcast 10.0.0.100 up
#/sbin/route add -host 10.0.0.100 dev eth0:0
/sbin/route add -host $MySQL_VIP dev eth0:0 #为VIP添加路由
/sbin/ipvsadm -A -t $MySQL_VIP:3306 -s wlc -p 3 #添加一个VIP
/sbin/ipvsadm -a -t $MySQL_VIP:3306 -r $MySQL_RIP1:3306 -g -w 1 #将VIP:3306的请求转发到RIP1上,-g表示使用DR模式,-w表示权重为1
/sbin/ipvsadm -a -t $MySQL_VIP:3306 -r $MySQL_RIP2:3306 -g -w 1
touch /var/lock/subsys/ipvsadm >/dev/null 2>&1 #新建文件ipvsadm,主要用来表示服务器已经启动,为status做准备
;;
stop)
/sbin/ipvsadm -C #清空LVS所有规则
/sbin/ipvsadm -Z #所有服务器计数器清零
ifconfig eth0:0 down #禁用虚拟网卡eth0:0
route del $MySQL_VIP #删除VIP的路由
rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1
echo "ipvsadm stoped"
;; #相当于C语言中的break,跳出case
status)
if [ ! -e /var/lock/subsys/ipvsadm ];then #如果ipvsadm不存在则输出ipvsadm stoped
echo "ipvsadm stoped"
exit 1
else
echo "ipvsadm OK"
fi
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0
 
五、配置LVS-DR-BACKUP
把以下所有文件拷到LVS-DR-BACKUP上
scp switchdr 10.0.0.162:/etc/init.d/
scp -r /etc/ha.d 10.0.0.162:/etc
scp /etc/sysconfig/ipvsadm 10.0.0.162:/etc/sysconfig/
然后注意备份节点的ha.cf文件在lvs-backup上把ha.cf中的
ucast eth0 10.0.0.133
六、配置成员机(MySQL1-Realserver 、MySQL2-Realserver) 脚本关闭arp
脚本如下:
!/bin/bash
# description: Config realserver lo and apply noarp
#Written by :NetSeek http://www.linuxtone.org
MySQL_VIP=10.0.0.100 #虚拟IP
/etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $MySQL_VIP netmask 255.255.255.255 broadcast $MySQL_VIP #新建虚拟ip,LVS中所有的VIP都一致
/sbin/route add -host $MySQL_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $MySQL_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
七、验证步骤
1、确认heartbeat 主从
[root@localhost ~]# ifconfig |grep inet
inet addr:10.0.0.133 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::2e0:4dff:fe99:3dbc/64 Scope:Link
inet addr:10.0.0.100 Bcast:10.0.0.100 Mask:255.255.255.255
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
可以确认到 10.0.0.100 LVS-DR-VIP 已经存在。从别的服务器也能ping通。
2,确认 LVS中设定的列表:
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.100:3306 wlc persistent 3
-> 10.0.0.152:3306 Route 1 0 0
-> 10.0.0.151:3306 Route 1 0 0
3、确认转发成功
[root@mgm yum.repos.d]# /usr/local/mysql/bin/mysql -u client -p -h 10.0.0.100
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5 to server version: 5.1.5-alpha-max
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> show databases;
--------------------
| Database |
--------------------
| information_schema |
| cc |
| cluster |
| huzi |
| mysql |
| ndb_1_fs |
| test |
--------------------
7 rows in set (0.01 sec)
mysql>
打开另一个shell连接数据库
[root@mgm yum.repos.d]# /usr/local/mysql/bin/mysql -u client -p -h 10.0.0.100
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5 to server version: 5.1.5-alpha-max
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show databases;
--------------------
| Database |
--------------------
| information_schema |
| cluster |
| huzi |
| mysql |
| test |
--------------------
5 rows in set (0.00 sec)
 
mysql>
查看LVS中设定的列表
[root@huzi ha.d]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.100:3306 wrr
-> 10.0.0.152:3306 Route 10 0 0
-> 10.0.0.151:3306 Route 0 0 0
查看连接状态
[root@localhost bin]# ipvsadm -L -n -c
IPVS connection entries
pro expire state source virtual destination
TCP 00:05 ESTABLISHED 10.0.0.162:36730 10.0.0.100:3306 10.0.0.151:3306
TCP 00:51 NONE 10.0.0.165:0 10.0.0.100:3306 10.0.0.152:3306
TCP 00:18 ESTABLISHED 10.0.0.165:33520 10.0.0.100:3306 10.0.0.152:3306
TCP 00:18 NONE 10.0.0.163:0 10.0.0.100:3306 10.0.0.152:3306
TCP 00:37 NONE 10.0.0.162:0 10.0.0.100:3306 10.0.0.151:3306
4、在MySQL客户端确认ldirector 定期来确认MySQL的连接情况:
[root@XXmdl19 var]# tcpdump host 100.XXX.3.87
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:37:39.427884 IP XXmdl19.mysql > XXlvs01.34356: S 3961650748:3961650748(0) ack 2091235633 win 5792
11:37:39.428437 IP XXmdl19.mysql > XXlvs01.34356: P 1:106(105) ack 2 win 46
11:37:39.428468 IP XXmdl19.mysql > XXlvs01.34356: F 106:106(0) ack 2 win 46
11:37:49.428994 IP XXmdl19.mysql > XXlvs01.34359: S 3986364821:3986364821(0) ack 2105720293 win 5792
11:37:49.429205 IP XXmdl19.mysql > XXlvs01.34359: P 1:106(105) ack 2 win 46
11:37:49.429236 IP XXmdl19.mysql > XXlvs01.34359: F 106:106(0) ack 2 win 46
11:37:59.430272 IP XXmdl19.mysql > XXlvs01.34362: S 3987864478:3987864478(0) ack 2110335516 win 5792
11:37:59.430462 IP XXmdl19.mysql > XXlvs01.34362: P 1:106(105) ack 2 win 46
11:37:59.430489 IP XXmdl19.mysql > XXlvs01.34362: F 106:106(0) ack 2 win 46
11:38:09.432210 IP XXmdl19.mysql > XXlvs01.34365: S 4007363487:4007363487(0) ack 2123944523 win 5792
11:38:09.432427 IP XXmdl19.mysql > XXlvs01.34365: P 1:106(105) ack 2 win 46
11:38:09.432456 IP XXmdl19.mysql > XXlvs01.34365: F 106:106(0) ack 2 win 46
 
12 packets captured
24 packets received by filter
0 packets dropped by kernel
可以确认是10秒中一次访问。
 
4,在访问端确认是否可以访问MYSQL,并且是不同的MySQL。
[root@XXfapp07 ~]# mysql -uXXfxs -pXXfxs -h10.0.0.100 -e "show variables like \"%pid%\"" ;
--------------- ----------------------------------
| Variable_name | Value |
--------------- ----------------------------------
| pid_file | /usr/local/mysql/var/Mysql2.pid |
--------------- ----------------------------------
[root@XXfapp07 ~]# mysql -uXXfxs -pXXfxs -h10.0.0.100 -e "show variables like \"%pid%\"" ;
--------------- ----------------------------------
| Variable_name | Value |
--------------- ----------------------------------
| pid_file | /usr/local/mysql/var/Mysql1.pid |
--------------- ----------------------------------
[root@XXfapp07 ~]#
5,重启LVS-DR-Master
[root@XXlvs01 ha.d]# reboot
Broadcast message from root (pts/0) (Wed Mar 25 11:42:19 2009):
The system is going down for reboot NOW!
在 36秒后 从LVS 进行接管:
在客户端进行确认:
[root@XXfapp07 ~]# mysql -uXXfxs -pXXfxs -h10.0.0.100 -e "show variables like \"%pid%\"" ;
--------------- ----------------------------------
| Variable_name | Value |
--------------- ----------------------------------
| pid_file | /usr/local/mysql/var/Mysql2.pid |
--------------- ----------------------------------
[root@XXfapp07 ~]# mysql -uXXfxs -pXXfxs -h10.0.0.100 -e "show variables like \"%pid%\"" ;
--------------- ----------------------------------
| Variable_name | Value |
--------------- ----------------------------------
| pid_file | /usr/local/mysql/var/Mysql1.pid |
--------------- ----------------------------------
[root@XXfapp07 ~]#
在切换期间无法访问,切换完毕后 又可以进行访问,并且轮询分配。
6,停止一台MySQL。
初始:
[root@XXlvs02 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RXXoteAddress:Port Forward Weight ActiveConn InActConn
TCP 100.XXX.3.81:mysql rr
-> XXmdl20:mysql Route 1 0 0
-> XXmdl19:mysql Route 1 0 0
-> XXmdl18:mysql Route 1 0 0
[root@XXlvs02 ~]#
停止MySQL XXmdl19
[root@XXlvs02 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RXXoteAddress:Port Forward Weight ActiveConn InActConn
TCP 100.XXX.3.81:mysql rr
-> XXmdl20:mysql Route 1 0 0
-> XXmdl18:mysql Route 1 0 0
[root@XXlvs02 ~]#
在 10秒内自动将 XXmdl19的 MySQL转发在 LVS中删除,只有两个了。
再访问MySQL时,能正常访问MySQL 20 和 18。
7、用NDB管理节点查看SQL节点连接状态
[root@mgm yum.repos.d]# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: mgm:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 @10.0.0.151 (Version: 5.1.5, Nodegroup: 0, Master)
id=2 (not connected, accepting connect from node2)
 
[ndb_mgmd(MGM)] 1 node(s)
id=3 @10.0.0.100 (Version: 5.1.5)
 
[mysqld(API)] 3 node(s)
id=4 @10.0.0.152 (Version: 5.1.5)
id=5 @10.0.0.151 (Version: 5.1.5)
id=6 (not connected, accepting connect from any host)
ndb_mgm>
标签: MySQL
扩展知识