手把手教你搭建zookeeper和kafka集群（超级详细）

一、环境准备

1、准备3台机器
主机名称主机IPzookeeper版本kafka版本worker01192.168.19.130zookeeper-3.6.3kafka_2.12-3.0.1worker02192.168.19.131zookeeper-3.6.3kafka_2.12-3.0.1worker03192.168.19.132zookeeper-3.6.3kafka_2.12-3.0.1
2、3台机器安装jdk1.8环境

3、下载kafka安装包（此处下载，可忽略第二步：下载安装包）:

 kafka_2.12-3.0.1.tgz

4、下载zookeeper安装包（此处下载，可忽略第二步：下载安装包）：

apache-zookeeper-3.6.3-bin.tar.gz

二、下载安装包（官方下载，仅供参考）

1、登录官方：Index of /

2、下载zookeeper安装包，apache-zookeeper-3.6.3-bin.tar.gz

3、下载kafka安装包，kafka_2.12-3.0.1.tgz

三、环境配置

1、修改主机名（分别单独执行）

# 修改主机名，在worker01上执行

hostnamectl set-hostname worker01

bash # 不重启，使修改主机名生效

# 修改主机名，在worker02上执行

hostnamectl set-hostname worker02

bash # 不重启，使修改主机名生效

# 修改主机名，在worker03上执行

hostnamectl set-hostname worker03

bash # 不重启，使修改主机名生效

2、关闭防火墙(worker01、worker02、worker02都执行)

systemctl stop firewalld

systemctl disable firewalld

systemctl status firewalld

3、配置/etc/hosts(worker01、worker02、worker02都执行)

cat >> /etc/hosts << EOF

192.168.19.130 worker01
192.168.19.131 worker02
192.168.19.132 worker03

EOF

四、安装 jdk

1、查询仓库jdk软件包(worker01、worker02、worker02都执行)

# 查询jdk软件包
yum list |grep jdk

# 安装系统对应版本
yum install -y java-1.8.0-openjdk-devel.x86_64

# 查看jdk是否配置生效
java -version

# 后续可以用jps查看，主机启动的所有java进程
jps

五、安装zookeeper集群

1、解压目录（worker01、worker02、worker03，都要执行）

# 创建目录
mkdir -p /home/kafaka-zookeeper

cd /home/kafaka-zookeeper

# 上传安装包apache-zookeeper-3.6.3-bin.tar.gz到该目录下

# 解压
tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz

# 重命名
mv apache-zookeeper-3.6.3-bin apache-zookeeper-3.6.3

2、修改配置文件（worker01、worker02、worker03，都执行）

# 创建目录
mkdir -p /home/kafaka-zookeeper/apache-zookeeper-3.6.3/{data,logs}

# 修改配置文件名
cd /home/kafaka-zookeeper/apache-zookeeper-3.6.3/conf

mv zoo_sample.cfg zoo.cfg

3、修改配置文件（worker01、worker02、worker03，都要执行）

vi zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/kafaka-zookeeper/apache-zookeeper-3.6.3/data
dataLogDir=/home/kafaka-zookeeper/apache-zookeeper-3.6.3/logs
clientPort=2181
server.1=192.168.19.130:2888:3888
server.2=192.168.19.131:2888:3888
server.3=192.168.19.132:2888:3888

4、创建myid

# 在 worker01、worker02、worker03 上都执行
cd /home/kafaka-zookeeper/apache-zookeeper-3.6.3/data

# 在 worker01、worker02、worker03 上都执行
touch myid

# 只在 192.168.19.130 worker01 上执行
echo 1 > /home/kafaka-zookeeper/apache-zookeeper-3.6.3/data/myid

# 只在192.168.19.131 worker02 上执行
echo 2 > /home/kafaka-zookeeper/apache-zookeeper-3.6.3/data/myid

# 只在192.168.19.132 worker03 上执行
echo 3 > /home/kafaka-zookeeper/apache-zookeeper-3.6.3/data/myid

5、zookeeper启动、停止

# zookeeper 启动
/home/kafaka-zookeeper/apache-zookeeper-3.6.3/bin/zkServer.sh start

# zookeeper 停止
/home/kafaka-zookeeper/apache-zookeeper-3.6.3/bin/zkServer.sh stop

# zookeeper 查看状态
/home/kafaka-zookeeper/apache-zookeeper-3.6.3/bin/zkServer.sh status

# 3台机器的状态
mode：follower    # 从
mode：leader      # 主
mode：follower    # 从

# 运行客户端（后面介绍kafka注册）
/home/kafaka-zookeeper/apache-zookeeper-3.6.3/bin/zkCli.sh -server worker0x的IP:2181

3台主机，需要全部执行启动命令，查看zookeeper启动状态，如果正常选主，就启动成功了；反之，失败，需要检查日志 dataLogDir=/home/kafaka-zookeeper/apache-zookeeper-3.6.3/logs。

六、安装kafka集群

1、解压目录（worker01、worker02、worker03，都要执行）

# 进入目录
cd /home/kafaka-zookeeper

# 上传kafka_2.12-3.0.1.tgz到该目录下

# 解压安装包kafka_2.12-3.0.1.tgz
tar -zxvf kafka_2.12-3.0.1.tgz

2、修改配置（worker01、worker02、worker03，都要执行）

# 创建目录
mkdir -p /home/kafaka-zookeeper/kafka_2.12-3.0.1/{data,logs}

# 进入kafka_2.12-3.0.1的配置文件目录
cd /home/kafaka-zookeeper/kafka_2.12-3.0.1/config

# 备份配置文件
cp server.properties server.properties.bak

# 删除注释行
sed -i "/#/d" server.properties

vim server.properties

# worker01 192.168.19.130 配置文件修改
broker.id=1
listeners=PLAINTEXT://192.168.19.130:9092
num.network.threads=12
num.io.threads=24
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/kafaka-zookeeper/kafka_2.12-3.0.1/logs
num.partitions=3
num.recovery.threads.per.data.dir=12
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connect=192.168.19.130:2181,192.168.19.131:2181,192.168.19.132:2181

zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

# 其它2台配置

# worker02 192.168.19.131 配置文件修改
broker.id=2
listeners=PLAINTEXT://192.168.19.131:9092
log.dirs=/home/kafaka-zookeeper/kafka_2.12-3.0.1/logs
num.partitions=3
zookeeper.connect=192.168.19.130:2181,192.168.19.131:2181,192.168.19.132:2181

# worker03 192.168.19.132 配置文件修改
broker.id=3
listeners=PLAINTEXT://192.168.19.132:9092
log.dirs=/home/kafaka-zookeeper/kafka_2.12-3.0.1/logs
num.partitions=3
zookeeper.connect=192.168.19.130:2181,192.168.19.131:2181,192.168.19.132:2181

3、启动kafka集群

# 启动kafka
nohup /home/kafaka-zookeeper/kafka_2.12-3.0.1/bin/kafka-server-start.sh /home/kafaka-zookeeper/kafka_2.12-3.0.1/config/server.properties &>> /home/kafaka-zookeeper/kafka_2.12-3.0.1/logs &

4、验证kafka集群

cd /home/kafaka-zookeeper/kafka_2.12-3.0.1/bin

# 创建 kafka topic hkzs
 ./kafka-topics.sh --create --bootstrap-server 192.168.19.130:9092,192.168.19.131:9092,192.168.19.132:9092 --replication-factor 3 --partitions 3 --topic zbqy

# 列出所有的topic
./kafka-topics.sh --list --bootstrap-server 192.168.19.130:9092

# 在worker01 192.168.19.130 上发布消息
./kafka-console-producer.sh --broker-list  192.168.19.130:9092 --topic hkzs
>我过的挺好
>zoo

#  在worker02 192.168.19.131 上消费
./kafka-console-consumer.sh --bootstrap-server 192.168.19.131:9092 --topic hkzs --from-beginning
我过的挺好
zoo

#  在worker02 192.168.19.132 上消费
./kafka-console-consumer.sh --bootstrap-server 192.168.19.132:9092 --topic hkzs --from-beginning
我过的挺好
zoo

5、验证zookeeper集群

cd /home/kafaka-zookeeper/apache-zookeeper-3.6.3/bin

# 进入zookeeper客户端，如果是自定义端口一定要 -server 指定IP:port，否则默认进入2181端口

./zkCli.sh -server 192.168.19.131:2181

# 查看服务
WatchedEvent state:SyncConnected type:None path:null
[zk: 192.168.19.131:2181(CONNECTED) 0] ls /brokers/ids
[1, 2, 3]
[zk: 192.168.19.131:2181(CONNECTED) 1] ls /brokers/topics 
[__consumer_offsets, hkzs]

# 进入zookeeper客户端，不是自定义
./zkCli.sh
[zk: localhost:2181(CONNECTED) 1] ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]
[zk: localhost:2181(CONNECTED) 2] ls /config 
[brokers, changes, clients, ips, topics, users]
[zk: localhost:2181(CONNECTED) 3] ls /config/brokers
[]
[zk: localhost:2181(CONNECTED) 4] ls /config/topics 
[__consumer_offsets, hkzs, zbqy]