Kafka三节点集群搭建

文章目录

1.集群规划

部署kafka集群最好是奇数台至少3台服务器，也可以在一台服务中部署三个kafka，修改端口号即可。
IP主机名部署的服务192.168.20.10kafka-1kafka+zookeeper192.168.20.11kafka-2kafka+zookeeper192.168.20.12kafka-3kafka+zookeeper

2.部署Zookeeper集群

2.1.在三台服务器中分别部署Zookeeper节点

1.安装JAVA环境
tar xf jdk-8u211-linux-x64.tar.gz -C /data/
vim /etc/profile
JAVA_HOME=/data/jdk1.8.0_211
PATH=$JAVA_HOME/bin:$PATH
source /etc/profile

2.安装Zookeeper
tar xf apache-zookeeper-3.5.8-bin.tar.gz -C /data/
mv /data/apache-zookeeper-3.5.8-bin/ /data/zookeeper_cluster

3.创建数据存储路径
mkdir /data/zookeeper_cluster/data

4.准备配置文件
cp /data/zookeeper_cluster/conf/zoo_sample.cfg  /data/zookeeper_cluster/conf/zoo.cfg

2.2.配置各个节点的数据存储路径

1.zookeeper-1节点
[root@zookeeper-1 ~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
dataDir=/data/zookeeper_cluster/data

2.zookeeper-1节点
[root@zookeeper-2 ~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
dataDir=/data/zookeeper_cluster/data

3.zookeeper-1节点
[root@kzookeeper-3~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
dataDir=/data/zookeeper_cluster/data

到此为止每个节点都配置完成了，下面进行集群模式的配置

2.3.将三节点配置成集群模式

1.为每个节点创建ServerID文件

1.zookeeper-1节点
[root@zookeeper-1 ~]# echo 1 > /data/zookeeper_cluster/data/myid

2.zookeeper-1节点
[root@zookeeper-2 ~]# echo 2 > /data/zookeeper_cluster/data/myid

3.zookeeper-1节点
[root@zookeeper-3 ~]# echo 3 > /data/zookeeper_cluster/data/myid

2.修改每个节点的配置文件增加集群各节点的通信地址

server.1=192.168.20.10:2888:3888

server.后面的数字就是每个节点创建的serverid号，2888是服务器之间的通信端口，3888是服务器之间的投票选举端口。

1.zookeeper-1节点
[root@kzookeeper-1~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888

2.zookeeper-2节点
[root@kzookeeper-2~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888

3.zookeeper-3节点
[root@kzookeeper-3~]# vim /data/zookeeper_cluster/conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888

2.4.启动Zookeeper集群

1.zookeeper-1节点
[root@zookeeper-1 ~]# /data/zookeeper_cluster/bin/zkServer.sh start

2.zookeeper-1节点
[root@zookeeper-2 ~]# /data/zookeeper_cluster/bin/zkServer.sh start

3.zookeeper-1节点
[root@zookeeper-3 ~]# /data/zookeeper_cluster/bin/zkServer.sh start

2.5.查看集群的状态

可以看到集群中zookeeper-1节点是Follower角色，zookeeper-2节点是Leader角色，zookeeper-3节点是Follower角色，正就验证了我们起那么说的那句话，当ServerID大并且投票超过半数则当选Leader节点，zookeeper-2节点启动后，ServerID最大，并且zookeeper-1和zookeeper-2都投了投票，票数也超过了集群数量的半数以上，因此当选成功。

1.zookeeper-1节点
[root@zookeeper-1 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

2.zookeeper-1节点
[root@zookeeper-2 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

3.zookeeper-1节点
[root@zookeeper-3 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

3.部署Kafka集群

3.1.在三台服务中部署Kafka

tar xf kafka_2.11-2.4.1.tgz -C /data/
mv /data/kafka_2.11-2.4.1/ /data/kafka

3.2.配置Kafka每个节点

kafka-1节点配置

[root@kafka-1 ~]# vim /data/kafka/config/server.properties 
broker.id=0                                        #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.10:9092           #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs                    #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181                #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

kafka-2节点配置

[root@kafka-2 ~]# vim /data/kafka/config/server.properties 
broker.id=1                                        #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.11:9092           #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs                    #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181                #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

kafka-3节点配置

[root@kafka-3 ~]# vim /data/kafka/config/server.properties 
broker.id=2                                    #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.12:9092           #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs                    #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181                #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

3.3.启动Kafka集群

[root@kafka-1 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
[root@kafka-2 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
[root@kafka-3 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties

4.在ZK集群中查看Kafka集群信息

在zookeeper集群中查看Kafka注册的broker信息，三个节点会在brokers子节点中有三个节点。

[root@kafka-1 ~]# /data/zookeeper_cluster/bin/zkCli.sh 
[zk: localhost:2181(CONNECTED) 3] ls /brokers/ids
[0, 1, 2]