linux安装kafka

linux安装kafka

官网地址

右键复制链接

[root@MeterSphere /usr/local]$wget https://downloads.apache.org/kafka/3.3.1/kafka_2.12-3.3.1.tgz

下载后解压cd

[root@MeterSphere /usr/local]$pwd
/usr/local
[root@MeterSphere /usr/local]$tar zxvf kafka_2.12-3.3.1.tgz

查看配置文件

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/config]$pwd
/usr/local/kafka_2.12-3.3.1/config
[root@MeterSphere /usr/local/kafka_2.12-3.3.1/config]$vim server.properties

下面的是日志地址，那就在这个地址下看看有没有对应的目录，没有的话就新建即可

其中broker.id应该是唯一的，在我看来就是kafka的身份证号
listeners 设置成内网ip
advertised.listener设置成公网ip
不可以配置成一样的，kafka启动的时候会报错
这两个字段的含义

查看内网地址，eth0的inet就是了

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.42.2  netmask 255.255.255.0  broadcast 172.16.42.255
        inet6 fe80::216:3eff:fe06:3053  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:06:30:53  txqueuelen 1000(Ethernet)
        RX packets 611381  bytes 111044418(105.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 556388  bytes 454959009(433.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000(Local Loopback)
        RX packets 55864  bytes 16903555(16.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55864  bytes 16903555(16.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

查看外网ip

[root@MeterSphere /home/test]$curl ifconfig.me
XX.XX.XXX.XXX[root@MeterSphere /home/test]$

比较重要的配置还有log.dirs

这里面存储的是kafka的各种消息，信息的磁盘持久化
还有num.partitions=1，就是一个broker有多少个partitions，默认的话就只有一个

zookeeper.connect 就是zookeeper的地址

环境变量

[root@MeterSphere /usr/local/kafka_2.12-3.3.1]$pwd
/usr/local/kafka_2.12-3.3.1
[root@MeterSphere /usr/local/kafka_2.12-3.3.1]$vim /etc/profile

新增如下内容

#KAFKA ENV
export KAFKA_HOME=/usr/local/kafka_2.12-3.3.1
export PATH=$PATH:$KAFKA_HOME/bin

配置立即生效

[root@MeterSphere /usr/local/kafka_2.12-3.3.1]$source /etc/profile

启动kafka

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/bin]$./kafka-server-start.sh  /usr/local/kafka_2.12-3.3.1/config/server.properties

停止kafka

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/bin]$./kafka-server-stop.sh

启动日志最后是这样的,开启的是9092端口

[2022-11-04 18:30:25,345] INFO Kafka version: 3.3.1 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-04 18:30:25,346] INFO Kafka commitId: e23c59d00e687ff5 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-04 18:30:25,346] INFO Kafka startTimeMs: 1667557825340 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-04 18:30:25,347] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2022-11-04 18:30:25,450] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use broker localhost:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2022-11-04 18:30:25,450] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use broker localhost:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2022-11-09 19:50:39,013] WARN [Controller id=0, targetBrokerId=0] Connection to node 0 (/47.95.210.129:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-11-09 19:50:39,013] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)

然后查看了controller的日志

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/logs]$tail -f controller.log
[2022-11-09 20:09:34,929] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker 47.95.210.129:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 47.95.210.129:9092 (id: 0 rack: null) failed.
        at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
        at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:292)
        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:246)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

复制这个47.95.210.129，也就是你配置的外网地址

[root@MeterSphere /etc]$vim hosts

在最后两行加上你刚才复制的地址即可

尝试了，不可以，所以把advertised.listener注释掉了，启动成功了，为什么，不知道
创建topic

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/bin]$ ./kafka-topics.sh --create --zookeeper 47.95.210.129:2181 --replication-factor 1--partitions 1 --topic xdclass-topic
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:396)
        at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

原因：kafka版本过高所致，2.2+=的版本，已经不需要依赖zookeeper来查看/创建topic，新版本使用 --bootstrap-server替换老版本的 --zookeeper-server
FUCK，还是不行

[root@MeterSphere /usr/local/kafka_2.12-3.3.1/bin]$ ./kafka-topics.sh --create --bootstrap-server 47.95.210.129:2181 --replication-factor 1 --partitions 1 --topic xdclass-topic
[2022-11-09 20:35:58,133] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/47.95.210.129:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)[2022-11-09 20:36:05,253] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/47.95.210.129:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)