0
0

KafKa 集群【docker compose】

文章目录

主机准备

IPcontroller idbroker id192.168.142.15715192.168.142.15626192.168.142.15537192.168.142.15848

部署

创建文件夹

  1. mkdir -p /docker/kafka/{data,data1,data2}
  1. chmod 777 -R {data,data1,data2}

编辑 docker-compose.yml

controller

docker-compose.yml

  1. services:
  2.   controller:
  3.     image: bitnami/kafka:3.8.0
  4.     container_name: controller
  5.     network_mode: host
  6.     restart: always
  7.     volumes:
  8.       - /docker/kafka/data:/data
  9.       - /etc/localtime:/etc/localtime:ro
  10.     environment:
  11.       KAFKA_CFG_NODE_ID: 1
  12.       KAFKA_CFG_PROCESS_ROLES: controller
  13.       KAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093
  14.       KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
  15.       KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  16.       KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
  17.       KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  18.       KAFKA_CFG_LOG_DIRS: /data

broker

  1. image: bitnami/kafka:3.8.0
  2.     container_name: broker
  3.     network_mode: host
  4.     restart: always
  5.     volumes:
  6.       - /etc/localtime:/etc/localtime:ro
  7.       - /docker/kafka/data1:/data1
  8.       - /docker/kafka/data2:/data2
  9.     environment:
  10.       KAFKA_CFG_NODE_ID: 5
  11.       KAFKA_CFG_PROCESS_ROLES: broker
  12.       KAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092
  13.       KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092
  14.       KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  15.       KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  16.       KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  17.       KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
  18.       KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  19.       KAFKA_CFG_LOG_DIRS: "/data1,/data2"

所有的主机这一段基本都是一样的 只有 id 不同，id 可以根据上面表格进行修改

生成cluster_id

  1. root@master:/docker/kafka# docker run -it --rm bitnami/kafka:3.8.0 kafka-storage.sh random-uuid
  2. kafka 06:43:36.79 INFO  ==>
  3. kafka 06:43:36.80 INFO  ==> Welcome to the Bitnami kafka container
  4. kafka 06:43:36.80 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
  5. kafka 06:43:36.80 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
  6. kafka 06:43:36.80 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
  7. kafka 06:43:36.80 INFO  ==>
  9. PavLImfXSqaOOdrMgq1teA

拿到 id

PavLImfXSqaOOdrMgq1teA

新增到 docker-compose.yml 文件中

一篇完整的 docker-compose.yml 文件

  1. services:
  2.   controller:
  3.     image: bitnami/kafka:3.8.0
  4.     container_name: controller
  5.     network_mode: host
  6.     restart: always
  7.     volumes:
  8.       - /docker/kafka/data:/data
  9.       - /etc/localtime:/etc/localtime:ro
  10.     environment:
  11.       KAFKA_CFG_NODE_ID: 1
  12.       KAFKA_CFG_PROCESS_ROLES: controller
  13.       KAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093
  14.       KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
  15.       KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  16.       KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
  17.       KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  18.       KAFKA_CFG_LOG_DIRS: /data
  19.       KAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teA
  20.   broker:
  21.     image: bitnami/kafka:3.8.0
  22.     container_name: broker
  23.     network_mode: host
  24.     restart: always
  25.     volumes:
  26.       - /etc/localtime:/etc/localtime:ro
  27.       - /docker/kafka/data1:/data1
  28.       - /docker/kafka/data2:/data2
  29.     environment:
  30.       KAFKA_CFG_NODE_ID: 5
  31.       KAFKA_CFG_PROCESS_ROLES: broker
  32.       KAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092
  33.       KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092
  34.       KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  35.       KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  36.       KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
  37.       KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
  38.       KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  39.       KAFKA_CFG_LOG_DIRS: "/data1,/data2"
  40.       KAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teA

启动

  1. docker compose up -d

查看启动状态

  1. docker ps -a | grep Up | wc -l
  2. 2

查看集群状态

使用 kafka-ui 查看

拉取 kafka-ui

只需要在一台主机上部署 ui 就可以了

  1. docker run -d -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui:v0.7.1

通过 ip + 8080 端口访问

在这里插入图片描述

添加集群

在这里插入图片描述
在这里插入图片描述

查看集群状态

在这里插入图片描述

使用命令行查看

  1. root@master:/docker/kafka# docker exec -it broker ./opt/bitnami/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller 192.168.142.157:9093,192.168.142.156:9093,192.168.142.155:9093,192.168.142.158:9093 describe --status
  2. ClusterId:              PavLImfXSqaOOdrMgq1teA
  3. LeaderId:               1
  4. LeaderEpoch:            22
  5. HighWatermark:          7695
  6. MaxFollowerLag:         0
  7. MaxFollowerLagTimeMs:   0
  8. CurrentVoters:          [1,2,3,4]
  9. CurrentObservers:       [5,6,7,8]

到此集群搭建完成

配置讲解

controller

参数解释KAFKA_CFG_NODE_ID节点 idKAFKA_CFG_PROCESS_ROLES节点角色KAFKA_CFG_LISTENERS节点监听的 ip 和端口，并且标记为 controllerKAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP定义不同监听器使用的加密协议KAFKA_CFG_CONTROLLER_LISTENER_NAMES控制器（Controller）用于接收来自其他控制器节点和 broker 节点请求的监听器（Listener）的名称KAFKA_CFG_CONTROLLER_QUORUM_VOTERS添加集群节点KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS当一个新的消费者加入到一个空的消费组时，协调者（Coordinator）将延迟多长时间再开始重新平衡（Rebalance）的过程KAFKA_CFG_LOG_DIRS日志目录KAFKA_KRAFT_CLUSTER_ID集群 id

broker

参数解释KAFKA_CFG_NODE_ID节点 idKAFKA_CFG_PROCESS_ROLES节点角色KAFKA_CFG_LISTENERS节点监听的 ip 和端口，并且标记为 brokerKAFKA_CFG_INTER_BROKER_LISTENER_NAME集群中 broker 之间通信所使用的监听器名称KAFKA_CFG_CONTROLLER_LISTENER_NAMES监听器的名称KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP定义不同监听器使用的加密协议KAFKA_CFG_CONTROLLER_QUORUM_VOTERS添加集群节点KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS当一个新的消费者加入到一个空的消费组时，协调者（Coordinator）将延迟多长时间再开始重新平衡（Rebalance）的过程KAFKA_CFG_LOG_DIRS日志目录KAFKA_KRAFT_CLUSTER_ID集群 id

标签： 中间件
本文转载自: https://blog.csdn.net/qq_62866151/article/details/142855971
版权归原作者 栀夏613 所有， 如有侵权，请联系我们删除。
登录后发布评论

“KafKa 集群【docker compose】”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？