Kafka监控EFAK(Kafka-eagle)部署与踩坑详细记录

环境

阿里云服务器centos
kafka 2.4.1 （1.1以上版本都能支持，低版本不太清楚了）
efak 3.0.1 当前时间官网下载的最新版（原名叫kafka-eagle）
efak官网：http://www.kafka-eagle.org/
jdk8

部署好的UI

在这里插入图片描述

部署流程

网上很多，我这总结下吧（除非你刚好和别人博客的环境一摸一样，否则大概率吃瘪，跑不顺利）

首先，

kafka-eagle

监控比较全面，但是，不知道是官网持续升级原因还是咋的，文档不太完整（个人看的是当前时间最新版本v3.0.1）

官网linux部署流程

efak官网：http://www.kafka-eagle.org/

去官网上看DOCS部分，

Install on Linux

照着步骤走完，如果你很顺利OK了（打开webUI，所有指标都正常监控起来了），那就不用往下看了

部署过程的命令行就不敲了，照着官方文档走就好，然后关于

efak

的配置文件我贴下我的（个人是单节点部署（kafka也是单节点））

配置文件几个关键点：

zookeeper的配置要和kafka里配置zk的路径一样（IP:port后的路径）

efak.zk.cluster.alias配置项配置哪几个集群别名，往后的配置就只需要配对应别名的即可

######################################
# multi zookeeper & kafka cluster list
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead
######################################
efak.zk.cluster.alias=cluster1
cluster1.zk.list=xdn10:2181
#cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181

######################################
# zookeeper enable acl
######################################
cluster1.zk.acl.enable=false
cluster1.zk.acl.schema=digest
cluster1.zk.acl.username=test
cluster1.zk.acl.password=test123

######################################
# broker size online list
######################################
cluster1.efak.broker.size=20

######################################
# zk client thread limit
######################################
kafka.zk.limit.size=16

######################################
# EFAK webui port
######################################
efak.webui.port=18048

######################################
# EFAK enable distributed
######################################
efak.distributed.enable=false
efak.cluster.mode.status=master
efak.worknode.master.host=localhost
efak.worknode.port=8085

######################################
# kafka jmx acl and ssl authenticate
######################################
cluster1.efak.jmx.acl=false
cluster1.efak.jmx.user=keadmin
cluster1.efak.jmx.password=keadmin123
cluster1.efak.jmx.ssl=false
cluster1.efak.jmx.truststore.location=/data/ssl/certificates/kafka.truststore
cluster1.efak.jmx.truststore.password=ke123456

######################################
# kafka offset storage
######################################
cluster1.efak.offset.storage=kafka
#cluster2.efak.offset.storage=zk

######################################
# kafka jmx uri
######################################
cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi

######################################
# kafka metrics, 15 days by default
######################################
efak.metrics.charts=true
efak.metrics.retain=7

######################################
# kafka sql topic records max
######################################
efak.sql.topic.records.max=5000
efak.sql.topic.preview.records.max=10

######################################
# delete kafka topic token
######################################
efak.topic.token=keadmin

######################################
# kafka sasl authenticate
######################################
cluster1.efak.sasl.enable=false
cluster1.efak.sasl.protocol=SASL_PLAINTEXT
cluster1.efak.sasl.mechanism=SCRAM-SHA-256
cluster1.efak.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka" password="kafka-eagle";
cluster1.efak.sasl.client.id=
cluster1.efak.blacklist.topics=
cluster1.efak.sasl.cgroup.enable=false
cluster1.efak.sasl.cgroup.topics=
cluster2.efak.sasl.enable=false
cluster2.efak.sasl.protocol=SASL_PLAINTEXT
cluster2.efak.sasl.mechanism=PLAIN
cluster2.efak.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-eagle";
cluster2.efak.sasl.client.id=
cluster2.efak.blacklist.topics=
cluster2.efak.sasl.cgroup.enable=false
cluster2.efak.sasl.cgroup.topics=

######################################
# kafka ssl authenticate
######################################
cluster3.efak.ssl.enable=false
cluster3.efak.ssl.protocol=SSL
cluster3.efak.ssl.truststore.location=
cluster3.efak.ssl.truststore.password=
cluster3.efak.ssl.keystore.location=
cluster3.efak.ssl.keystore.password=
cluster3.efak.ssl.key.password=
cluster3.efak.ssl.endpoint.identification.algorithm=https
cluster3.efak.blacklist.topics=
cluster3.efak.ssl.cgroup.enable=false
cluster3.efak.ssl.cgroup.topics=

######################################
# kafka sqlite jdbc driver address
######################################
#efak.driver=org.sqlite.JDBC
#efak.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
#efak.username=root
#efak.password=www.kafka-eagle.org

######################################
# kafka mysql jdbc driver address
######################################
efak.driver=com.mysql.cj.jdbc.Driver
#efak.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.url=jdbc:mysql://rm-wz9976qu64y5dzypjno.mysql.rds.aliyuncs.com/ke?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowMultiQueries=true
efak.username=1111
efak.password=1111

部署注意点

以下不清楚原作者是否会逐步完善他的部署文档，如果已经完善的或者优化了代码，那可能就不存在了

kafka服务端要开启JMX

这里重点几个配置：开启JMX、设置JMX远程连接端口、设置JMX的服务绑定IP、设置JMX远程RMI通信端口

这几个配置分别对应：

-Dcom.sun.management.jmxremote-Dcom.sun.management.jmxremote.port=9999-Djava.rmi.server.hostname=172.26.117.190
-Dcom.sun.management.jmxremote.rmi.port=9998

大部分博客只有前两项配置，他们的环境网络比较通畅，如果你们服务器网络所有端口都互通，那么
-Dcom.sun.management.jmxremote.rmi.port=9998
不用配，因为是如果你不配，就会是随机端口

ip不用区分内网和外网ip，只要客户端能访问的通就行
端口也没有固定，只是参考。
端口配置有些博客或文章会写在kafka-server启动的那个脚本里JMX_PORT=9999，效果一样，也可以的

配置方式

jmx的配置在
kafka-run-class.sh
里

有些博客会要配置到
kafka-server-start.sh
其实也可以，但我为了统一一个地方，就放
kafka-run-class.sh
里了

修改

kafka-run-class.sh

（1）找到JMX settings位置# JMX settingsif[-z"$KAFKA_JMX_OPTS"];thenKAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.26.117.190"fi> 配置> > -Djava.rmi.server.hostname=172.26.117.190> > 是某些docker跑kafka的服务，如果不配置，没有绑定到指定ip上会外部无法连接
（2）找到JMX port to use位置> 开头注释的是原先配置。修改的逻辑是：判断是否是kafkaserver（这里插一句，改的是> > kafka-run-class.sh> > 文件，但是我们启动kafkaserver、运行命令行都会依赖调用到这个文件，shell熟悉的话去看一眼kafkaserver启动文件就知道了），如果是kafkaserver才把那几个参数设置上，否则当使用命令行操作kafka时，会出现端口被占用的报错，因为它也尝试去启动jmx服务# JMX port to use#if [ $JMX_PORT ]; then# KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "#fiJMX_PORT=9999JMX_RMI_PORT=9998ISKAFKASERVER="false"if[["$*"=~"kafka.Kafka"]];thenISKAFKASERVER="true"fiif[$JMX_PORT]&&["true"=="$ISKAFKASERVER"];thenKAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT -Dcom.sun.management.jmxremote.rmi.port=$JMX_RMI_PORT "echoset KAFKA_JMX_PORT:$KAFKA_JMX_OPTSfi

网络检查

上述涉及的所有IP、端口，都要互通，telnet要能连上。ping不通不重要，因为某些服务器会禁止ping，这个属于正常。

检查防火墙

1.查看防火墙状态：active (running) 即是开启状态systemctl status firewalld
2.查看已开发端口命令firewall-cmd --list-all
3.新增防火墙开放端口：firewall-cmd --zone=public --add-port=3306/tcp --permanent
4.开放端口后需要重新加载防火墙：firewall-cmd --reload
5.firewalld的基本使用命令：启动： systemctl start firewalld关闭： systemctl stop firewalld查看状态： systemctl status firewalld开机禁用： systemctl disable firewalld开机启用： systemctl enable firewalld

配置安全组

如果是买的云服务器，去配置安全组开放对应端口

注意，内网IP也要单独配置开放，不会因为配了外网可访问就认为内网也可以访问

常见问题

efak服务启动后webUI访问空白或访问不了

原因：表结构未创建

如果你用的是sqlite数据库，你可能也不会碰上此问题

如果你的mysql的jdbc.url是类似（IP:端口）
jdbc:mysql://127.0.0.1:3306/ke
然后还没有时区问题的，那么你大可能不会碰上此问题

我估计原作者习惯用sqlite或者本地mysql进行测试，比较顺利。但是如果使用RDS或其他云服务器等形式上的Mysql，服务启动就会出问题，而且，，，，说实话，你不知道服务已经有问题，因为没有相关日志

先讲下他服务启动会做什么：花了点时间看他源码

此问题的改动PR提交：[Fix-637][efak-plugin] Fixed mysql plugin auto create databases and tables

不想改代码的可直接拉取该改动的分支代码再重新编译即可

@ComponentpublicclassStartupListenerimplementsApplicationContextAware{privatestaticApplicationContext applicationContext;@OverridepublicvoidsetApplicationContext(ApplicationContext arg0)throwsBeansException{ContextSchema context =newContextSchema();
        context.start();}publicstaticObjectgetBean(String beanName){if(applicationContext ==null){
            applicationContext =ContextLoader.getCurrentWebApplicationContext();}return applicationContext.getBean(beanName);}publicstatic<T>TgetBean(String beanName,Class<T> clazz){return clazz.cast(getBean(beanName));}classContextSchemaextendsThread{publicvoidrun(){String jdbc =SystemConfigUtils.getProperty("efak.driver");if(JConstants.MYSQL_DRIVER_V5.equals(jdbc)||JConstants.MYSQL_DRIVER_V8.equals(jdbc)){MySqlRecordSchema.schema();}elseif(JConstants.SQLITE_DRIVER.equals(jdbc)){SqliteRecordSchema.schema();}initKafkaMetaData();}privatevoidinitKafkaMetaData(){KafkaCacheUtils.initKafkaMetaData();}}}

建立数据库和对应表结构：如果已存在就跳过。MySqlRecordSchema.schema();
初始化kafka元信息：从zk里读取。initKafkaMetaData();
（上方代码没有）定时器初始化：使用的是quartz，主要包括两类，一个是指标定时获取，配置的是1分钟1次；指标清理，每天1点，根据开放给我们的配置文件的指标保留时长配置项（默认15天）去清理指标数据（毕竟1分钟一次，而且存入数据库，不清理是会爆炸的）。【注意：定时器定时规则不开放配置，是写死在quartz配置文件里的】

坑在哪里？

建立数据库和对应表时，它拆解jdbc的url，就是原本咱们是例如：efak.driver=com.mysql.cj.jdbc.Driverefak.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&serverTimezone=Asia/Shanghaiefak.username=rootefak.password=123456他会拆解url，提取出 host、port然后丢弃后面一长串的参数（这是它自动创建数据库表时一个插件步骤，efak正式监控服务不会有此操作），这么做的考虑应该是连数据库ke也会自动create。但是这样的过程会有几个问题：- 第一：部分mysql版本会报时区不对导致无法创建mysql连接，就是serverTimezone配置项这项没有，会建立连接失败；连带着的是后面的所有连接参数都放弃了，如果某些mysql服务开了ssl，那更连不上> 这个问题某些服务环境下连报错都不会给你，开的异步匿名线程，异常信息被吞掉了。。。。。我还是跑源码才看到的- 第二：从如上源码可知，mysql数据库初始化如果失败了，那initKafkaMetaData();这行代码就不会被执行，那么后续所有和jmx的rmi调用和读取kafka流程都会报错以上，当然有能正常执行的场景，尤其本地的时候### 解决****1.个人建议，手动创建数据库和表。因为你要通过它自动化创建的话，要调整的地方还比较多，还要兼顾有些jdbcurl是域名形式的，域名形式的没有端口，那它读端口的代码你也得改，总之，个人尝试了下去改，改动地方还比较多，然后能在本地跑通，而跑RDS时，某些sql语句（内置的）还执行不了：我遇到的是建立索引的alter语句，最后还是手动执行了。手动执行怎么弄呢？他的SQL都在代码里写死列举出来了，V3.0.1版本的sql表结构初始化语句在本文最后****2.对源码中如下代码进行try-catchtry{if(JConstants.MYSQL_DRIVER_V5.equals(jdbc)||JConstants.MYSQL_DRIVER_V8.equals(jdbc)){MySqlRecordSchema.schema();}elseif(JConstants.SQLITE_DRIVER.equals(jdbc)){SqliteRecordSchema.schema();}}catch(Exception e){LoggerUtils.print(StartupListener.class).info("db schema init error, efak.driver:{}", jdbc, e);}3.重新打包> 打包不了，或者报什么source1.5啥的，在最外层pom文件加入如下配置<properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><java.version>1.8</java.version></properties><build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><configuration><source>${java.version}</source><target>${java.version}</target><encoding>${project.build.sourceEncoding}</encoding></configuration></plugin></plugins></build>> 最好执行跳过测试的打包，也就是去执行build.sh这个文件里的命令：> > mvn clean package -DskipTests> > > > 否则它会去尝试连接zk和kafka，因为有很多test类.打包好后，web工程target目录下的ke.war替换到部署目录里/usr/local/efak/kms/webapps目录下多说一句，源码日志真的很少，，，，，

efak服务UI里Node>kafka 这个页面内存和CPU以及Version展示异常

Kafka version is “-” or JMX Port is “-1” maybe kafka broker jmxport disable.
The memory and cpu is null. may be the kafka broker down or blocked.

这是webUI的提示，这种大概率是没开启JMX服务或者JMX访问不通（这个可以在efak的服务日志里看到连接xxx失败或拒绝连接）。导致原因也就是要么kafka配置JMX问题，要么就是网络不通，照着上述部署点去检查

日志报错：Telnet [localhost:8085] has crash, please check it

这是你开了efak分布式部署配置，但实际并未分布式部署，此端口不通

此问题只是efak自身分部署通信不通问题，单节点部署就不用管了，不影响监控kafka，或者你去查查代码

我这边即使efak.distributed.enable=false 也会一直有这个报错，因该是代码有点问题。可以先不管

日志报错：ERROR - JMX service url[xxxx:9999] create has error,msg is java.lang.NullPointerException

JMX远程连接不通，检查kafka端jmx配置和网络是否通畅，照着上述部署点去检查

日志报错：Get kafka os memory from jmx has error, msg is null

JMX远程连接不通，检查kafka端jmx配置和网络是否通畅，照着上述部署点去检查

日志报错：Get topic size from jmx has error, msg is null

JMX远程连接不通，检查kafka端jmx配置和网络是否通畅，照着上述部署点去检查

日志报错：java.rmi.ConnectException: Connection refused to host: xxxx;

JMX远程连接不通，检查kafka端jmx配置和网络是否通畅，照着上述部署点去检查

附：mysql的ke数据库的表创建语句

记得先创建一个名为 ke 的数据库

如下表创建语句在源码中都有

SET NAMES utf8mb4;SET FOREIGN_KEY_CHECKS =0;-- ------------------------------ Table structure for ke_alarm_clusters-- ----------------------------DROPTABLEIFEXISTS`ke_alarm_clusters`;CREATETABLE`ke_alarm_clusters`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`type`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`server`textCHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL,`alarm_group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`alarm_times`int(0)NULLDEFAULTNULL,`alarm_max_times`int(0)NULLDEFAULTNULL,`alarm_level`varchar(4)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`is_normal`varchar(2)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULT'Y',`is_enable`varchar(2)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULT'Y',`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`modify`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_alarm_config-- ----------------------------DROPTABLEIFEXISTS`ke_alarm_config`;CREATETABLE`ke_alarm_config`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`alarm_group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`alarm_type`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`alarm_url`textCHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL,`http_method`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`alarm_address`textCHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL,`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`modify`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`cluster`,`alarm_group`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_alarm_consumer-- ----------------------------DROPTABLEIFEXISTS`ke_alarm_consumer`;CREATETABLE`ke_alarm_consumer`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`topic`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`lag`bigint(0)NULLDEFAULTNULL,`alarm_group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`alarm_times`int(0)NULLDEFAULTNULL,`alarm_max_times`int(0)NULLDEFAULTNULL,`alarm_level`varchar(4)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`is_normal`varchar(2)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULT'Y',`is_enable`varchar(2)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULT'Y',`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`modify`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_alarm_crontab-- ----------------------------DROPTABLEIFEXISTS`ke_alarm_crontab`;CREATETABLE`ke_alarm_crontab`(`id`bigint(0)NOTNULL,`type`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`crontab`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`is_enable`varchar(2)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULT'Y',`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`modify`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`,`type`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_connect_config-- ----------------------------DROPTABLEIFEXISTS`ke_connect_config`;CREATETABLE`ke_connect_config`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`connect_uri`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`version`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`alive`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`modify`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_consumer_bscreen_press-- ----------------------------DROPTABLEIFEXISTS`ke_consumer_bscreen_press`;CREATETABLE`ke_consumer_bscreen_press`(`id`bigintunsignedNOTNULL,`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`topic`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`logsize`bigint(0)NULLDEFAULTNULL,`difflogsize`bigint(0)NULLDEFAULTNULL,`offsets`bigint(0)NULLDEFAULTNULL,`diffoffsets`bigint(0)NULLDEFAULTNULL,`lag`bigint(0)NULLDEFAULTNULL,`timespan`bigint(0)NULLDEFAULTNULL,`tm`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`)USINGBTREE,INDEX`idx_timespan`(`timespan`)USINGBTREE,INDEX`idx_tm_cluster_diffoffsets`(`tm`,`cluster`,`diffoffsets`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=1CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_consumer_group-- ----------------------------DROPTABLEIFEXISTS`ke_consumer_group`;CREATETABLE`ke_consumer_group`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`topic`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`status`int(0)NULLDEFAULTNULL,PRIMARYKEY(`cluster`,`group`,`topic`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_consumer_group_summary-- ----------------------------DROPTABLEIFEXISTS`ke_consumer_group_summary`;CREATETABLE`ke_consumer_group_summary`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`group`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`topic_number`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`coordinator`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`active_topic`int(0)NULLDEFAULTNULL,`active_thread_total`int(0)NULLDEFAULTNULL,PRIMARYKEY(`cluster`,`group`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_logsize-- ----------------------------DROPTABLEIFEXISTS`ke_logsize`;CREATETABLE`ke_logsize`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`topic`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`logsize`bigint(0)NULLDEFAULTNULL,`diffval`bigint(0)NULLDEFAULTNULL,`timespan`bigint(0)NULLDEFAULTNULL,`tm`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,INDEX`idx_timespan`(`timespan`)USINGBTREE,INDEX`idx_tm_topic`(`tm`,`topic`)USINGBTREE,INDEX`idx_tm_cluster_diffval`(`tm`,`cluster`,`diffval`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_metrics-- ----------------------------DROPTABLEIFEXISTS`ke_metrics`;CREATETABLE`ke_metrics`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`broker`textCHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL,`type`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`key`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`value`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`timespan`bigint(0)NULLDEFAULTNULL,`tm`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,INDEX`idx_tm_cluster_all`(`cluster`,`type`,`key`,`timespan`,`tm`)USINGBTREE,INDEX`idx_tm_cluster_key`(`cluster`,`type`,`key`,`tm`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_metrics_offline-- ----------------------------DROPTABLEIFEXISTS`ke_metrics_offline`;CREATETABLE`ke_metrics_offline`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`key`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`one`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`mean`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`five`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`fifteen`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`cluster`,`key`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_p_role-- ----------------------------DROPTABLEIFEXISTS`ke_p_role`;CREATETABLE`ke_p_role`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`name`varchar(64)CHARACTERSET utf8 COLLATE utf8_general_ci NOTNULLCOMMENT'role name',`seq`tinyint(0)NOTNULLCOMMENT'rank',`description`varchar(128)CHARACTERSET utf8 COLLATE utf8_general_ci NOTNULLCOMMENT'role describe',PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=4CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Records of ke_p_role-- ----------------------------INSERTINTO`ke_p_role`VALUES(1,'Administrator',1,'Have all permissions');INSERTINTO`ke_p_role`VALUES(2,'Devs',2,'Own add or delete');INSERTINTO`ke_p_role`VALUES(3,'Tourist',3,'Only viewer');-- ------------------------------ Table structure for ke_resources-- ----------------------------DROPTABLEIFEXISTS`ke_resources`;CREATETABLE`ke_resources`(`resource_id`bigint(0)NOTNULLAUTO_INCREMENT,`name`varchar(255)CHARACTERSET utf8 COLLATE utf8_general_ci NOTNULLCOMMENT'resource name',`url`varchar(255)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`parent_id`int(0)NOTNULL,PRIMARYKEY(`resource_id`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=25CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Records of ke_resources-- ----------------------------INSERTINTO`ke_resources`VALUES(1,'System','/system',-1);INSERTINTO`ke_resources`VALUES(2,'User','/system/user',1);INSERTINTO`ke_resources`VALUES(3,'Role','/system/role',1);INSERTINTO`ke_resources`VALUES(4,'Resource','/system/resource',1);INSERTINTO`ke_resources`VALUES(5,'Notice','/system/notice',1);INSERTINTO`ke_resources`VALUES(6,'Topic','/topic',-1);INSERTINTO`ke_resources`VALUES(7,'Message','/topic/message',6);INSERTINTO`ke_resources`VALUES(8,'Create','/topic/create',6);INSERTINTO`ke_resources`VALUES(9,'Alarm','/alarm',-1);INSERTINTO`ke_resources`VALUES(10,'Add','/alarm/add',9);INSERTINTO`ke_resources`VALUES(11,'Modify','/alarm/modify',9);INSERTINTO`ke_resources`VALUES(12,'Cluster','/cluster',-1);INSERTINTO`ke_resources`VALUES(13,'ZkCli','/cluster/zkcli',12);INSERTINTO`ke_resources`VALUES(14,'UserDelete','/system/user/delete',1);INSERTINTO`ke_resources`VALUES(15,'UserModify','/system/user/modify',1);INSERTINTO`ke_resources`VALUES(16,'Mock','/topic/mock',6);INSERTINTO`ke_resources`VALUES(18,'Create','/alarm/create',9);INSERTINTO`ke_resources`VALUES(19,'History','/alarm/history',9);INSERTINTO`ke_resources`VALUES(20,'Manager','/topic/manager',6);INSERTINTO`ke_resources`VALUES(21,'PasswdReset','/system/user/reset',1);INSERTINTO`ke_resources`VALUES(22,'Config','/alarm/config',9);INSERTINTO`ke_resources`VALUES(23,'List','/alarm/list',9);INSERTINTO`ke_resources`VALUES(24,'Hub','/topic/hub',6);-- ------------------------------ Table structure for ke_role_resource-- ----------------------------DROPTABLEIFEXISTS`ke_role_resource`;CREATETABLE`ke_role_resource`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`role_id`int(0)NOTNULL,`resource_id`int(0)NOTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=26CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Records of ke_role_resource-- ----------------------------INSERTINTO`ke_role_resource`VALUES(1,1,1);INSERTINTO`ke_role_resource`VALUES(2,1,2);INSERTINTO`ke_role_resource`VALUES(3,1,3);INSERTINTO`ke_role_resource`VALUES(4,1,4);INSERTINTO`ke_role_resource`VALUES(5,1,5);INSERTINTO`ke_role_resource`VALUES(6,1,7);INSERTINTO`ke_role_resource`VALUES(7,1,8);INSERTINTO`ke_role_resource`VALUES(8,1,10);INSERTINTO`ke_role_resource`VALUES(9,1,11);INSERTINTO`ke_role_resource`VALUES(10,1,13);INSERTINTO`ke_role_resource`VALUES(11,2,7);INSERTINTO`ke_role_resource`VALUES(12,2,8);INSERTINTO`ke_role_resource`VALUES(13,2,13);INSERTINTO`ke_role_resource`VALUES(14,2,10);INSERTINTO`ke_role_resource`VALUES(15,2,11);INSERTINTO`ke_role_resource`VALUES(16,1,14);INSERTINTO`ke_role_resource`VALUES(17,1,15);INSERTINTO`ke_role_resource`VALUES(18,1,16);INSERTINTO`ke_role_resource`VALUES(19,1,18);INSERTINTO`ke_role_resource`VALUES(20,1,19);INSERTINTO`ke_role_resource`VALUES(21,1,20);INSERTINTO`ke_role_resource`VALUES(22,1,21);INSERTINTO`ke_role_resource`VALUES(23,1,22);INSERTINTO`ke_role_resource`VALUES(24,1,23);INSERTINTO`ke_role_resource`VALUES(25,1,24);-- ------------------------------ Table structure for ke_sql_history-- ----------------------------DROPTABLEIFEXISTS`ke_sql_history`;CREATETABLE`ke_sql_history`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`username`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`host`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`ksql`textCHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL,`status`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`spend_time`bigint(0)NULLDEFAULTNULL,`created`varchar(32)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,`tm`varchar(16)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULLDEFAULTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_topic_rank-- ----------------------------DROPTABLEIFEXISTS`ke_topic_rank`;CREATETABLE`ke_topic_rank`(`cluster`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`topic`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`tkey`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`tvalue`bigint(0)NULLDEFAULTNULL,PRIMARYKEY(`cluster`,`topic`,`tkey`)USINGBTREE)ENGINE=InnoDBCHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Table structure for ke_user_role-- ----------------------------DROPTABLEIFEXISTS`ke_user_role`;CREATETABLE`ke_user_role`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`user_id`int(0)NOTNULL,`role_id`tinyint(0)NOTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=2CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Records of ke_user_role-- ----------------------------INSERTINTO`ke_user_role`VALUES(1,1,1);-- ------------------------------ Table structure for ke_users-- ----------------------------DROPTABLEIFEXISTS`ke_users`;CREATETABLE`ke_users`(`id`bigint(0)NOTNULLAUTO_INCREMENT,`rtxno`int(0)NOTNULL,`username`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`password`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`email`varchar(64)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,`realname`varchar(128)CHARACTERSET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOTNULL,PRIMARYKEY(`id`)USINGBTREE)ENGINE=InnoDBAUTO_INCREMENT=2CHARACTERSET= utf8mb4 COLLATE= utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;-- ------------------------------ Records of ke_users-- ----------------------------INSERTINTO`ke_users`VALUES(1,1000,'admin','123456','[email protected]','Administrator');-- ------------------------------ View structure for ke_topic_consumer_group_summary_view-- ----------------------------DROPVIEWIFEXISTS`ke_topic_consumer_group_summary_view`;CREATEALGORITHM=UNDEFINEDDEFINER=`root`@`localhost`SQL SECURITY DEFINERVIEW`ke_topic_consumer_group_summary_view`ASselect`ke_consumer_group`.`cluster`AS`cluster`,`ke_consumer_group`.`topic`AS`topic`,count(distinct`ke_consumer_group`.`group`)AS`group_number`,count(distinct`ke_consumer_group`.`group`)AS`active_group`from`ke_consumer_group`where(`ke_consumer_group`.`status`=0)groupby`ke_consumer_group`.`cluster`,`ke_consumer_group`.`topic`;SET FOREIGN_KEY_CHECKS =1;

标签： kafka EFAK kafka-eagle

本文转载自: https://blog.csdn.net/maoyuanming0806/article/details/127248506
版权归原作者 小明同学YYDS 所有，如有侵权，请联系我们删除。

Kafka监控EFAK(Kafka-eagle)部署与踩坑详细记录

环境

部署好的UI

部署流程

官网linux部署流程

部署注意点

kafka服务端要开启JMX

网络检查

检查防火墙

配置安全组

常见问题

efak服务启动后webUI访问空白或访问不了

原因：表结构未创建

efak服务UI里Node>kafka 这个页面内存和CPU以及Version展示异常

日志报错：Telnet [localhost:8085] has crash, please check it

日志报错：ERROR - JMX service url[xxxx:9999] create has error,msg is java.lang.NullPointerException

日志报错：Get kafka os memory from jmx has error, msg is null

日志报错：Get topic size from jmx has error, msg is null

日志报错：java.rmi.ConnectException: Connection refused to host: xxxx;

附：mysql的ke数据库的表创建语句

发表评论

“Kafka监控EFAK(Kafka-eagle)部署与踩坑详细记录”的评论:

关于作者

overfit同步小助手

相关阅读

文章导航