0


Docker 搭建 zookeeper、kafka 集群

1、zookeeper 搭建

首先创建一个自定义网络,后续的所有容器都放入同一个内网中,容器之间还可以通过容器名称进行直接访问,在后续的配置中只需要写明容器名称即可,会自动找到对应的IP地址,防止重启容器后IP地址发生变化时,还要去修改配置文件的操作

docker network create kafka-net

1、zookeeper 单机搭建

创建目录

mkdir-p /mnt/data/zookeeper/data # 数据挂载目录mkdir-p /mnt/data/zookeeper/conf # 配置挂载目录mkdir-p /mnt/data/zookeeper/logs # 日志挂载目录

启动zookeeper

docker pull zookeeper:3.5.9

docker run -d--privileged=true --name zookeeper -p2181:2181 --network kafka-net --network-alias zookeeper -v /mnt/data/zookeeper/data:/data -v /mnt/data/zookeeper/conf:/conf -v /mnt/data/zookeeper/logs:/datalog zookeeper:3.5.9

2888为组成zookeeper服务器之间的通信端口,3888为用来选举leader的端口

进入zookeeper容器

dockerexec-it 容器id /bin/bash

找到配置文件位置

cat /conf/zoo.cfg

查看zookeeper状态

zkServer.sh status

2、zookeeper 集群搭建

  • 注意,此处是在一台服务器上搭建的伪集群,以下是docker compose的部署方案
  • 注意:安装时遇到的问题,zookeeper 的版本必须是 3.5.0 以后,3.5.0 版本之前的配置与 3.5.0 版本之后是不同的,选错版本会导致 zookeeper 无法形成集群,最终导致使用 kafka 时报错!!!!!!
mkdir-p /mnt/data/zookeeper
cd /mnt/data/zookeeper
vim zookeeper-compose.yml

zookeeper-compose.yml 内容如下

version:'3'services:zoo1:image: zookeeper:3.5.9
    container_name: zoo1
    restart: always
    privileged:trueports:- 2182:2181environment:ZOO_MY_ID:1ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:image: zookeeper:3.5.9
    container_name: zoo2
    restart: always
    privileged:trueports:- 2183:2181environment:ZOO_MY_ID:2ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:image: zookeeper:3.5.9
    container_name: zoo3
    restart: always
    privileged:trueports:- 2184:2181environment:ZOO_MY_ID:3ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

networks:default:external:truename: kafka-net
  • 注意:docker-compose会默认将当前的容器名称作为network的alias
  • 执行 docker-compose up,表示执行当前路径下的yml文件。也可以指定执行某一个yml文件 - docker-compose -f zookeeper-compose.yml up -d
  • 通过 docker inspect 容器ID 查看容器的网络配置

2、kafka 搭建

1、docer搭建kafka集群

docker pull wurstmeister/kafka
docker run -d--privileged=true \--name kafka0 -p9092:9092 \--network kafka-net \
--network-alias kafka0 \-eKAFKA_BROKER_ID=0\-eKAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \-eKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \-eKAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9092 \-eALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest

docker run -d--privileged=true \--name kafka1 -p9093:9092 \--network kafka-net \
--network-alias kafka1 \-eKAFKA_BROKER_ID=1\-eKAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \-eKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \-eKAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9093 \-eALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest

docker run -d--privileged=true \--name kafka2 -p9094:9092 \--network kafka-net \
--network-alias kafka2 \-eKAFKA_BROKER_ID=2\-eKAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 \-eKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \-eKAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.231.10:9094 \-eALLOW_PLAINTEXT_LISTENER=yes \
wurstmeister/kafka:latest
  • KAFKA_LISTENERS 与 KAFKA_ADVERTISED_LISTENERS 是为了区分内网和外网的
  • 如果只有内网访问,就可以只配置 KAFKA_LISTENERS
  • 如果涉及了外网访问,比如要在云服务器上部署使用,就需要配置 KAFKA_ADVERTISED_LISTENERS 参数了,在本文开始时配置了一个 kafka-net 的网络,也就是处于这个网络下的访问才属于内网访问,而 kafka 部署完毕后,需要在其他服务器上访问 kafka,这就需要通过外网访问 kafka,所以必须配置 KAFKA_ADVERTISED_LISTENERS ,且值为 PLAINTEXT://<服务器ip>:<暴露端口> ,例如我的服务器外网地址为 192.168.231.10,端口为容器暴露的端口
  • KAFKA_BROKER_ID : broker的ID,这个ID是集群的标识,不能重复
  • KAFKA_ZOOKEEPER_CONNECT:zookeeper的连接地址
  • KAFKA_LISTENERS:标识kafka服务运行在容器内的9092端口,因为没有指定host,所以是0.0.0.0标识所有的网络接口
  • KAFKA_ADVERTISED_LISTENERS:kafka发布到zookeeper供客户端使用的服务地址

2、docker compose 搭建kafka集群

  • 注意,此处是在一台服务器上搭建的伪集群,以下是docker compose的部署方案
  • 注意,需要等待 zookeeper 完全启动好后再启动 kafka 集群
mkdir-p /mnt/data/kafka/node1/data # 数据挂载目录mkdir-p /mnt/data/kafka/node2/data # 数据挂载目录mkdir-p /mnt/data/kafka/node3/data # 数据挂载目录cd /mnt/data/kafka
vim kafka-compose.yml
version:'3'services:broker1:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker1
    privileged:trueports:-"9092:9092"environment:KAFKA_BROKER_ID:1KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9092## 宿主机IPKAFKA_ADVERTISED_PORT:9092## zookeeper 集群,因为都在同一个内网,所以可以通过容器名称获取到容器的ipKAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node1/data/:/kafka"broker2:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker2
    privileged:trueports:-"9093:9092"environment:KAFKA_BROKER_ID:2KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9093## 宿主机IPKAFKA_ADVERTISED_PORT:9093KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node2/data/:/kafka"broker3:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker3
    privileged:trueports:-"9094:9092"environment:KAFKA_BROKER_ID:3KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9094## 宿主机IPKAFKA_ADVERTISED_PORT:9094KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node3/data/:/kafka"networks:default:external:truename: kafka-net
  • 执行 docker-compose up,表示执行当前路径下的yml文件。也可以指定执行某一个yml文件 - docker-compose -f kafka-compose.yml up -d
  • 通过 docker inspect 容器ID 查看容器的网络配置

3、docker compose 整体启动

前面可以省略,直接创建一个docker-compose.yml 文件即可启动 3台 zookeeper 与 3台 kafka

version:'3'services:zoo1:image: zookeeper:3.5.9
    container_name: zoo1
    restart: always
    ports:- 2182:2181environment:ZOO_MY_ID:1ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:image: zookeeper:3.5.9
    container_name: zoo2
    restart: always
    ports:- 2183:2181environment:ZOO_MY_ID:2ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:image: zookeeper:3.5.9
    container_name: zoo3
    restart: always
    ports:- 2184:2181environment:ZOO_MY_ID:3ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  broker1:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker1
    ports:-"9092:9092"# depends_on:通过 depends_on 告诉 docker-compose 当前服务启动之前先要把 depends_on 指定的服务启动起来才行# 例如:kafka 依赖 zookeeperdepends_on:- zoo1
      - zoo2
      - zoo3
    environment:KAFKA_BROKER_ID:1KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9092## 宿主机IPKAFKA_ADVERTISED_HOST_NAME: broker1
      KAFKA_ADVERTISED_PORT:9092KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node1/data/:/kafka"broker2:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker2
    ports:-"9093:9092"depends_on:- zoo1
      - zoo2
      - zoo3
    environment:KAFKA_BROKER_ID:2KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9093## 宿主机IPKAFKA_ADVERTISED_HOST_NAME: broker2
      KAFKA_ADVERTISED_PORT:9093KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node2/data/:/kafka"broker3:image: wurstmeister/kafka:latest
    restart: always
    container_name: broker3
    ports:-"9094:9092"depends_on:- zoo1
      - zoo2
      - zoo3
    environment:KAFKA_BROKER_ID:3KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.231.10:9094## 宿主机IPKAFKA_ADVERTISED_HOST_NAME: broker3
      KAFKA_ADVERTISED_PORT:9094KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181volumes:-"/mnt/data/kafka/node3/data/:/kafka"networks:default:external:truename: kafka-net

4、可视化工具 offsetexplorer

在这里插入图片描述
在这里插入图片描述

5、可视化工具 kafka-map

docker run -d\-p8080:8080 \-v /opt/kafka-map/data:/usr/local/kafka-map/data \-eDEFAULT_USERNAME=admin \-eDEFAULT_PASSWORD=admin \--name kafka-map \--restart always dushixiang/kafka-map:latest
容器启动后,访问:ip:8080 访问

kafka-map官网:https://gitee.com/dushixiang/kafka-map
标签: 运维 docker kafka

本文转载自: https://blog.csdn.net/weixin_43559374/article/details/131588024
版权归原作者 只因哥哥 所有, 如有侵权,请联系我们删除。

“Docker 搭建 zookeeper、kafka 集群”的评论:

还没有评论