Kafka-3.3.4摒弃zookeeper集群部署操作文档

一、集群规划

kafka从2.8版本开始，就允许用户在不需要zookeeper的情况下运行。真正全面摒弃zookeeper的版本是3.0，在后面的版本kafka使用了新的元数据管理方式kraft,提高了Kafka的可扩展性、可用性和性能。

下载地址：Apache Kafka

这里下载kafka_2.13-3.4.0.tgz版本，自带有支持raft模式
主机名称IPprocess.rolesnode.idmaster192.168.20.101broker,controller1salve1192.168.20.102broker,controller2salve2192.168.20.103broker,controller3

二、集群部署

将安装包上传到/opt/software目录下,解压到安装目录/opt/moudle/

[appview@master software]$ll 
-rw-rw-r--   1 appview appview 106290956 5月   2 08:26 kafka_2.13-3.4.0.tgz
[appview@master software]$ tar -zxvf kafka_2.13-3.4.0.tgz -C /opt/moudle/
[appview@master moudle]$ mv kafka_2.13-3.4.0  kafka-3.4.0
[appview@master kafka-3.4.0]$ cd config/kraft/
[appview@master kraft]$ ll
总用量 24
-rw-r--r-- 1 appview appview 6136 2月   1 2023 broker.properties
-rw-r--r-- 1 appview appview 5765 2月   1 2023 controller.properties
-rw-r--r-- 1 appview appview 6340 2月   1 2023 server.properties
[appview@master kraft]$ pwd
/opt/moudle/kafka-3.4.0/config/kraft
[appview@master kraft]$ vim server.properties

下面的文件是我的配置

只需要修改下面这几个配置成相应的ip地址以及node.id

node.id=1   （##修改）  ### node.id=2  ### node.id=3 对应我们的集群规划
[email protected]:9093,[email protected]:9093,[email protected]:9093
listeners=PLAINTEXT://192.168.20.101:9092,CONTROLLER://192.168.20.101:9093 （##修改）
advertised.listeners=PLAINTEXT://192.168.20.166:9092 （##修改）
log.dirs=/tmp/kraft-combined-logs （可选择）

配置文件server.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
​
#
# This configuration file is intended for use in KRaft mode, where
# Apache ZooKeeper is not present.  See config/kraft/README.md for details.
#
​
############################# Server Basics #############################
​
# The role of this server. Setting this puts us in KRaft mode
process.roles=broker,controller
​
# The node id associated with this instance's roles
node.id=1
​
# The connect string for the controller quorum
[email protected]:9093,[email protected]:9093,[email protected]:9093
​
############################# Socket Server Settings #############################
​
# The address the socket server listens on.
# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.20.101:9092,CONTROLLER://192.168.20.101:9093
​
# Name of listener used for communication between brokers.
inter.broker.listener.name=PLAINTEXT
​
# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://192.168.20.101:9092
​
# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.names=CONTROLLER
​
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
​
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
​
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
​
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
​
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
​
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
​
​
############################# Log Basics #############################
​
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kraft-combined-logs
​
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
​
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
​
############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
​
############################# Log Flush Policy #############################
​
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
​
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
​
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
​
############################# Log Retention Policy #############################
​
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
​
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
​
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
​
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
​
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

配置环境变量

[appview@master ~]$ vim .bashrc
[appview@master ~]$ pwd
/home/appview
[appview@master ~]$ ll /home/appview/.bashrc
-rw-r--r-- 1 appview appview 868 4月  13 22:35 /home/appview/.bashrc
#### 添加以下内容
#KAFKA_HOME
export KAFKA_HOME=/opt/moudle/kafka-3.4.0
export PATH=$PATH:$KAFKA_HOME/bin
​
#### source以下使环境变量生效
[appview@master ~]$ source /home/appview/.bashrc

分发安装包，没有分发脚本的可以scp过去，注意：分发之后记得配置其他机器的环境变量以及修改配置

[appview@master ~]$ xsync /opt/moudle/kafka-3.4.0/

三、集群ID初始化

切换到kafka-3.3.4家目录下执行以下命令生成集群唯一的ID，在其中一个节点执行即可

[appview@master kafka-3.4.0]$ ./bin/kafka-storage.sh random-uuid
rBEaIR-6Qx2xLfHh95nFCA
[appview@master kafka-3.4.0]$

指定集群id进行集群初始化，每个节点都执行

[appview@master kafka-3.4.0]$ ./bin/kafka-storage.sh format -t rBEaIR-6Qx2xLfHh95nFCA -c ./config/kraft/server.properties
Formatting /opt/moudle/kafka-3.4.0/logs/kraft-combined-logs with metadata.version 3.4-IV0.
[appview@master kafka-3.4.0]$

四、启动集群

在每个部署的节点执行以下启动命令，jps查看进程，三个节点都已经有进程了

[appview@master kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties
[appview@master kafka-3.4.0]$ jps
13577 Jps
13502 Kafka
[appview@master kafka-3.4.0]$
​
[appview@salve1 kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties
[appview@salve1 kafka-3.4.0]$ jps
12772 Jps
12669 Kafka
[appview@salve1 kafka-3.4.0]$
​
[appview@salve2 kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties
[appview@salve2 kafka-3.4.0]$ jps
6611 Jps
6591 Kafka
[appview@salve2 kafka-3.4.0]$

关闭集群命令

[appview@salve2 kafka-3.4.0]$ bin/kafka-server-stop.sh

五、集群验证

使用以下命令测试集群是否可用，如果和下面的一样就说明我们安装可以了

###### 列出所有主题
[appview@master kafka-3.4.0]$ bin/kafka-topics.sh --list --bootstrap-server master:9092
​
[appview@master kafka-3.4.0]$
###### 创建主题
[appview@master kafka-3.4.0]$ bin/kafka-topics.sh --create --bootstrap-server master:9092 --replication-factor 1 --partitions 1 --topic hello-world
Created topic hello-world.
[appview@master kafka-3.4.0]$
​
#### 启动生产者
[appview@master kafka-3.4.0]$ bin/kafka-console-producer.sh --bootstrap-server  master:9092  --topic hello-world
>
>
>how are you ?
>i am fine.
>
​
#### 启动salve1消费者验证
[appview@salve1 kafka-3.4.0]$ bin/kafka-console-consumer.sh --bootstrap-server salve1:9092  --topic hello-world --from-beginning
​
​
how are you ?
i am fine.
​
#### 启动salve2消费者验证
[appview@salve2 kafka-3.4.0]$ bin/kafka-console-consumer.sh --bootstrap-server salve2:9092  --topic hello-world --from-beginning
​
how are you ?
i am fine.
asdasdsafswfwqfqwgwg

六、结语

最近忙得天昏地暗的，没时间继续更新文档，kafka集群是我在实际项目中部署了多次总结的，其中最近的一个项目中测试环境部署了5套kafka集群，生产环境部署了2套，今天有空在自己的虚拟机上安装部署并把每个操作记录后分享的