kafka消息大小默认为1M,当需要调整消息大小时,可通过以下方式配置,亲测有效。
一、Kafka服务端配置文件修改
注意:kafka集群各个节点配置都要同步修改
(1)server.properties
1.修改以下配置:
# The send buffer (SO_SNDBUF) used by the socket server#socket.send.buffer.bytes=102400socket.send.buffer.bytes=10485760# The receive buffer (SO_RCVBUF) used by the socket server#socket.receive.buffer.bytes=102400socket.receive.buffer.bytes=10485760# The maximum size of a request that the socket server will accept (protection against OOM)socket.request.max.bytes=104857602.添加以下配置
#5242880(5M)6291456(6M) 10485760(10M) 125829120(120M)message.max.bytes=10485760#每个分区试图获取的消息字节数。要大于等于message.max.bytesreplica.fetch.max.bytes=12582912#原有的socket.send.buffer.bytes,socket.receive.buffer.bytes,socket.request.max.bytes也要改成和message.max.bytes一样的大小
(2)producer.properties
1. 修改以下配置
#请求的最大大小为字节9437184(9M)。要小于 message.max.bytes=10485760(10M)max.request.size=9437184
(3)consumer.properties
1. 添加以下配置
#每个提取请求中为每个主题分区提取的消息字节数。要大于等于message.max.bytes=10485760(10M)12582912(12M)fetch.message.max.bytes=12582912
(4)重启kafka
1. 先关闭kafka
./bin/kafka-server-stop.sh
2. 再启动kafka
bin/kafka-server-start.sh -daemon config/server.properties
二、客户端程序配置
##5242880 5M 10485760 10Mspring.kafka.producer.properties.max.request.size=10485760
spring.kafka.consumer.fetch-max-bytes=62914560
三、报错信息
当kafka需要发送大于1M的消息时,客户端程序必须配置这两个参数,否则将会报以下错误:
报错1:
`org.apache.kafka.common.errors.RecordTooLargeException:The message is xxxxx bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.`
报错2:
Kafka - Received invalid metadata error in produce request on partition. NetworkException when send large data to kafka topic
版权归原作者 IT小太阳2022 所有, 如有侵权,请联系我们删除。