kafka3.6.0集群部署

环境准备

• 机器环境系统主机名IP地址centos7.9kafka01192.168.200.51centos7.9kafka02192.168.200.52centos7.9kafka03192.168.200.53
• 所需软件jdk-8u171-linux-x64.tar.gzapache-zookeeper-3.8.3-bin.tar.gzhttps://dlcdn.apache.org/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gzkafka_2.13-3.6.0.tgzhttps://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz
• hosts设置192.168.200.51 kafka01192.168.200.52 kafka02192.168.200.53 kafka03
• java环境设置mkdir /data/javaln -s /data/java/ /opt/cd /opt/javatar -zxvf jdk-8u171-linux-x64.tar.gz -C /opt/java/``````cat /etc/profile.d/java.sh #javaexport JAVA_HOME=/opt/java/jdk1.8.0_171export JRE_HOME=$JAVA_HOME/jreexport CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATHexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH``````source /etc/profilejava -version

zookeeper安装部署

• 创建软件安装目录mkdir /data/server
• 解压安装tar -zxvf apache-zookeeper-3.8.3-bin.tar.gz -C /data/server/ln -s /data/server/apache-zookeeper-3.8.3-bin/ /data/server/zookeeper# 创建zookeeper数据存储目录mkdir /data/server/zookeeper/zkData# 创建zookeeper标识文件echo '1' > /data/server/zookeeper/zkData/myid
• 修改配置cp /data/server/zookeeper/zoo_sample.cfg /data/server/zookeeper/zoo.cfg``````vim /data/server/zookeeper/zoo.cfg# 修改数据存储目录dataDir=/data/server/zookeeper/zkData# 添加下列集群配置server.1=kafka01:2888:3888server.2=kafka02:2888:3888server.3=kafka03:2888:3888
• 分发软件# 分发软件至kafka02scp -r /data/server/apache-zookeeper-3.8.3-bin/ kafka02:/data/server/# 分发软件至kafka03scp -r /data/server/apache-zookeeper-3.8.3-bin/ kafka03:/data/server/
• kafka02与kafka03软链接ln -s /data/server/apache-zookeeper-3.8.3-bin/ /data/server/zookeeper
• kafka02与kafka03修改myid# kafka02修改myidecho '2' > /data/server/zookeeper/zkData/myid# kafka03修改myidecho '3' > /data/server/zookeeper/zkData/myid
• 防火墙放行端口firewall-cmd --zone=public --permanent --add-port={2181,2888,3888}/tcpfirewall-cmd --reload
• 设置开机自启vim /etc/systemd/system/zookeeper.service[Unit]Description=zookeeper serviceAfter=network.target [Service]User=rootGroup=root Type=forkingEnvironment="JAVA_HOME=/opt/java/jdk1.8.0_171"ExecStart=/data/server/zookeeper/bin/zkServer.sh startExecStop=/data/server/zookeeper/bin/zkServer.sh stopPrivateTmp=falseRestart=always [Install]WantedBy=multi-user.target``````systemctl daemon-reloadsystemctl start zookeepersystemctl enable zookeeper
• zookeeper状态查看- kafka01[root@kafka01 ~]# jps4049 Jps3899 QuorumPeerMain[root@kafka01 ~]# /data/server/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /data/server/zookeeper/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower- kafka02[root@kafka02 zookeeper]# jps2705 QuorumPeerMain2920 Jps[root@kafka02 zookeeper]# /data/server/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /data/server/zookeeper/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: follower- kafka03[root@kafka03 zookeeper]# jps3187 Jps3030 QuorumPeerMain[root@kafka03 ~]# /data/server/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /data/server/zookeeper/bin/../conf/zoo.cfgClient port found: 2181. Client address: localhost. Client SSL: false.Mode: leader

kafka安装部署

• 解压安装tar -zxvf kafka_2.13-3.6.0.tgz -C /data/server/ln -s /data/server/kafka_2.13-3.6.0/ /data/server/kafka
• 修改配置vim /data/server/kafka/config/server.propertiesbroker.id=0# 日志目录log.dirs=/data/server/kafka/kafka-logs# zookeeper连接地址及目录zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181/kafka
• 分发软件# 分发软件至kafka02scp -r /data/server/kafka_2.13-3.6.0/ kafka02:/data/server/分发软件至kafka03scp -r /data/server/kafka_2.13-3.6.0/ kafka03:/data/server/
• kafka02与kafka03软链接ln -s /data/server/kafka_2.13-3.6.0/ /data/server/kafka
• 修改kafka02和kafka03的broker.idvim /data/server/kafka/config/server.properties# kafka02broker.id=1# kafka03broker.id=2
• 防火墙放行端口firewall-cmd --zone=public --permanent --add-port=9092/tcpfirewall-cmd --reload
• 开机自启vim /etc/systemd/system/kafka.service``````[Unit]Description=kafka serviceAfter=network.target [Service]User=rootGroup=root Type=forkingEnvironment="JAVA_HOME=/opt/java/jdk1.8.0_171"ExecStart=/data/server/kafka/bin/kafka-server-start.sh -daemon /data/server/kafka/config/server.propertiesExecStop=/data/server/kafka/bin/kafka-server-stop.sh PrivateTmp=falseRestart=always [Install]WantedBy=multi-user.target``````systemctl daemon-reloadsystemctl start kafkasystemctl enable kafka
• 最终效果- kafka01[root@kafka01 ~]# jps5444 Kafka8661 Jps3899 QuorumPeerMain- kafka02[root@kafka02 ~]# jps2705 QuorumPeerMain3538 Jps3373 Kafka- kafka03[root@kafka03 ~]# jps3668 Kafka3030 QuorumPeerMain3896 Jps