当zookeeper启动失败：Invalid config, exiting abnormally

章节目录

环境说明

1. zookeepe镜像版本：wurstmeister/zookeeper latest 3f43f72cb283 3 years ago 510MB

问题说明

我有一次 zookeeper 容器启动过程中，不小心强制关机了，所以可能破坏了 zookeeper 某些数据，导致启动失败

zookeeper 启动报错日志

ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.13/bin/../conf/zoo.cfg
mkdir: cannot create directory '': No such file or directory
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.quorum.QuorumPeerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Invalid config, exiting abnormally

解决方案

zookeeper 容器名称是：

kafka-docker_zookeeper_1

1. 查看该容器挂载目录

docker inspect kafka-docker_zookeeper_1

[{"Mounts":[{"Type":"volume","Name":"ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02","Source":"/var/lib/docker/volumes/ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02/_data","Destination":"/opt/zookeeper-3.4.13/conf","Driver":"local","Mode":"rw","RW":true,"Propagation":""},{"Type":"volume","Name":"9690de948c110633badd453f11e9bca3fb3692e88f54ae40f9e726ad0c64332a","Source":"/var/lib/docker/volumes/9690de948c110633badd453f11e9bca3fb3692e88f54ae40f9e726ad0c64332a/_data","Destination":"/opt/zookeeper-3.4.13/data","Driver":"local","Mode":"rw","RW":true,"Propagation":""}],}]

2. 查看该容器挂载目录的配置文件

进入容器挂载目录

# 进入容器挂载目录cd /var/lib/docker/volumes/ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02/_data

目录文件如下所示：

[root@centos _data]# ll
总用量 4
-rw-r--r--. 1501ftp5356月  302018 configuration.xsl
-rw-r--r--. 1501ftp012月 1215:07 log4j.properties
-rw-r--r--. 1501ftp012月 1215:07 zoo.cfg

发现三个文件中，

log4j.properties

和

zoo.cfg

两个文件大小都是

0

，两个配置文件是空的，查看

zoo.cfg

内容，发现确实是空的，猜测 Zookeeper 启动失败正是因为配置文件的问题，需要补全。

3. 尝试查找旧的完整配置文件

zookeeper 以前有正常启动过的，都会在以下目录动态生成挂载目录，所以，在该目录下查找

zoo.cfg

文件，根据时间，找到以前生成的

zoo.cfg

配置文件

find /var/lib/docker/volumes/ -name zoo.cfg

[root@jiewli _data]# find /var/lib/docker/volumes/ -name zoo.cfg
/var/lib/docker/volumes/ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02/_data/zoo.cfg
/var/lib/docker/volumes/97b1950a9806f29eb4e4e81eb76ca2bd5d7700a72bc74bbb4324b13c446db701/_data/zoo.cfg
/var/lib/docker/volumes/7e9b3a353466085395fd3a2a4bede7b974a0ae8101ee3d87f1e15c1c17d36a38/_data/zoo.cfg
/var/lib/docker/volumes/64162e6eb0c30bc890fe07ac3acd60f8331131a5d8b5e55467d965e5573ee626/_data/zoo.cfg

找到了 4 个，找到其中日期为以前的：

[root@jiewli _data]# cd /var/lib/docker/volumes/7e9b3a353466085395fd3a2a4bede7b974a0ae8101ee3d87f1e15c1c17d36a38/_data[root@jiewli _data]# ll
总用量 12
-rw-r--r--. 1501ftp5356月  302018 configuration.xsl
-rw-r--r--. 1501ftp21616月  302018 log4j.properties
-rw-r--r--. 1501ftp9341月  202019 zoo.cfg

查看

zoo.cfg

内容：

cat /var/lib/docker/volumes/7e9b3a353466085395fd3a2a4bede7b974a0ae8101ee3d87f1e15c1c17d36a38/_data/zoo.cfg

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial # synchronization phase can takeinitLimit=10# The number of ticks that can pass between # sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just # example sakes.dataDir=/opt/zookeeper-3.4.13/data
# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the # administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1

查看

log4j.properties

内容：

cat /var/lib/docker/volumes/7e9b3a353466085395fd3a2a4bede7b974a0ae8101ee3d87f1e15c1c17d36a38/_data/log4j.properties

# Define some default values that can be overridden by system propertieszookeeper.root.logger=INFO, CONSOLE
zookeeper.console.threshold=INFO
zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log
zookeeper.log.threshold=DEBUG
zookeeper.tracelog.dir=.
zookeeper.tracelog.file=zookeeper_trace.log

## ZooKeeper Logging Configuration## Format is "<default threshold> (, <appender>)+# DEFAULT: console appender onlylog4j.rootLogger=${zookeeper.root.logger}# Example with rolling log file#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE# Example with rolling log file and tracing#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE## Log INFO level and above messages to the console#log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold}log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601}[myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n

## Add ROLLINGFILE to rootLogger to get log file output#    Log DEBUG level and above messages to a log filelog4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=${zookeeper.log.threshold}log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}# Max log file size of 10MBlog4j.appender.ROLLINGFILE.MaxFileSize=10MB
# uncomment the next line to limit number of backup files#log4j.appender.ROLLINGFILE.MaxBackupIndex=10log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601}[myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n

## Add TRACEFILE to rootLogger to get log file output#    Log DEBUG level and above messages to a log filelog4j.appender.TRACEFILE=org.apache.log4j.FileAppender
log4j.appender.TRACEFILE.Threshold=TRACE
log4j.appender.TRACEFILE.File=${zookeeper.tracelog.dir}/${zookeeper.tracelog.file}log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
### Notice we are including log4j's NDC here (%x)log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601}[myid:%X{myid}] - %-5p [%t:%C{1}@%L][%x] - %m%n

查看

configuration.xsl

内容：

cat /var/lib/docker/volumes/7e9b3a353466085395fd3a2a4bede7b974a0ae8101ee3d87f1e15c1c17d36a38/_data/configuration.xsl

<?xml version="1.0"?><xsl:stylesheetxmlns:xsl="http://www.w3.org/1999/XSL/Transform"version="1.0"><xsl:outputmethod="html"/><xsl:templatematch="configuration"><html><body><tableborder="1"><tr><td>name</td><td>value</td><td>description</td></tr><xsl:for-eachselect="property"><tr><td><aname="{name}"><xsl:value-ofselect="name"/></a></td><td><xsl:value-ofselect="value"/></td><td><xsl:value-ofselect="description"/></td></tr></xsl:for-each></table></body></html></xsl:template></xsl:stylesheet>

4. 用旧的完整配置文件覆盖疑似有问题的当前配置文件

三个文件，哪个有问题就覆盖哪个，我直接三个文件全部删掉，复制旧文件到当前挂载目录，如果没有旧文件，我已经将内容全部贴出来了，可以手动写入覆盖。

rm-f /var/lib/docker/volumes/ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02/_data/*

cp /var/lib/docker/volumes/64162e6eb0c30bc890fe07ac3acd60f8331131a5d8b5e55467d965e5573ee626/_data/* /var/lib/docker/volumes/ee425b5732918a15e1f3da9c9fb617d93b47cc0c67086c34005fa18e94f60e02/_data/

5. 重启容器

docker restart kafka-docker_zookeeper_1

正常

思维扩展，类似问题排查

所有启动时报

Invalid config, exiting abnormally

错误的问题，都可以按照上面的步骤排查配置文件问题，这次是

zoo.cfg

，或许也有可能是别的配置文件出问题。但只要按照上面步骤，也算是比较高效的排查方式。