1. kibana启动报错，kibana_system用户认证

[2022-09-06T19:47:01.491-04:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes..version%2Cnodes..http.publish_address%2Cnodes.*.ip]

解决办法：设置kibana_system用户的密码。
[es@goya1 cert]$ elasticsearch-reset-password -u kibana_system -i
warning: ignoring JAVA_HOME=/app/elasticsearch/jdk; using bundled JDK
This tool will reset the password of the [kibana_system] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y
Enter password for [kibana_system]: changeit123
Re-enter password for [kibana_system]: changeit123
Password for the [kibana_system] user successfully reset.

修改,确认kibana.yml文件的配置部分：

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

elasticsearch.username: "kibana_system"
elasticsearch.password: "changeit123"

2. logstash 启动报错

[2022-09-13T22:41:21,452][ERROR][logstash.configmanagement.bootstrapcheck] There are config files (1) in the '/app/logstash/config/tomcat_log_es.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config
ERROR: There are config files (1) in the '/app/logstash/config/tomcat_log_es.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2022-09-13T22:41:21,463][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at app.logstash.lib.bootstrap.environment.<main>(/app/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

解决办法：原因是logstash.yml配置了xpack相关的参数。开启X-Pack Management功能后，启动logstsh的时候就不用再配置logstash.conf文件了，启动的时候也不用再使用-f指定这个文件进行启动了一旦启动了logstash的集中管理，我们就可以直接启动logstash，而不用跟任何的参数。Logstash集中管理，先启动logstash，然后再设置相关配置。

要么关闭x-Pack，要么不用-f的参数启动，相应的配置在logstash.yml里进行。

3， logstash配置的logstash_admin_user修改密码报错

[es@goya1 head-master]$ elasticsearch-reset-password -u logstash_admin_user
warning: ignoring JAVA_HOME=/app/elasticsearch/jdk; using bundled JDK
This tool will reset the password of the [logstash_admin_user] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

ERROR: Failed to reset password for the [logstash_admin_user] user

解决办法：原因是因为ES里没有这个用户，可以尝试通过kibana控制台创建一个角色和logstash_admin_user用户来解决问题。参考：Logstash8.4在Linux系统上的安装以及配置Tomcat日志(ELK安装part2)(未完待续)_yangkei的博客-CSDN博客

4，Metricbeat启动后报错connection refused

{"log.level":"info","@timestamp":"2022-09-16T03:36:06.216-0400","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-16T03:36:06.217-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://192.168.88.5:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-16T03:36:14.259-0400","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.status: error making http request: Get "http://localhost:5601/api/status\": dial tcp [::1]:5601: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
解决办法：因为http://localhost:5601这个配置是从metricsbeat目录下的kibana.yml文件读取的，因此需要修改localhost为对应的ip地址。

[es@goya1 modules.d]$ pwd
/app/metricbeat/modules.d
[es@goya1 modules.d]$ cat kibana.yml

Module: kibana

Docs: https://www.elastic.co/guide/en/beats/metricbeat/main/metricbeat-module-kibana.html

• module: kibana
  #metricsets:

  - status

  period: 10s
  hosts: ["192.168.88.5:5601"]
  #basepath: ""
  #username: "user"
  #password: "secret"

5， Metricbeat启动报错

Exiting: loading configs: 3 errors: invalid config: config file ("/app/metricbeat/modules.d/kibana.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/kibana.yml'); invalid config: config file ("/app/metricbeat/modules.d/logstash.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/logstash.yml'); invalid config: config file ("/app/metricbeat/modules.d/system.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /app/metricbeat/modules.d/system.yml')
问题解决：metricbeat文件权限不能给太大，不然启动检测不能他通过，可以修改到750的权限。

6，Metricbeat监控tomcat日志报错

{"log.level":"info","@timestamp":"2022-09-20T10:58:01.529+0800","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://192.168.88.7:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-20T10:58:10.511+0800","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: error making http request: Get "http://localhost:9600/\": dial tcp [::1]:9600: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-20T10:58:10.512+0800","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node: error making http request: Get "http://localhost:9600/\": dial tcp 127.0.0.1:9600: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
问题解决：我使用了logstash提取tomcat日志，因此这里需要在Metricsbeat里再配置一下logstash.yml文件。

[es@goya1 modules.d]$ pwd
/app/metricbeat/modules.d
[es@goya1 modules.d]$ grep 9600 *
logstash-xpack.yml.disabled: hosts: ["localhost:9600"]
logstash.yml: hosts: ["localhost:9600"]

7，Filebeat启动报错

[es@goya1 filebeat]$ ./filebeat -e -c filebeat.kafka.yml

{"log.level":"error","@timestamp":"2022-09-26T16:56:43.325+0800","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: 1 error: setting 'filebeat.prospectors' has been removed","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: 1 error: setting 'filebeat.prospectors' has been removed
解决办法：在6.3版本以后，在配置文件中需要把filebeat.prospectors 修改为filebeat.inputs

[es@goya1 filebeat]$ cat filebeat.kafka.yml
filebeat.prospectors:

• type: log
  enabled: true
  paths:
  - /app/tomcat/logs/tomcat_access_json.2022-09-23.log

[es@goya1 filebeat]$ cat filebeat.kafka.yml
filebeat.inputs:

• type: log
  enabled: true
  paths:
  - /app/tomcat/logs/tomcat_access_json.2022-09-23.log