Prometheus实战篇:Prometheus监控rabbitmq

Prometheus实战篇:Prometheus监控rabbitmq

准备环境

docker-compose安装rabbitmq

这里注意rabbitmq需要暴露2个端口

docker-compose.yaml

version:'3'services:rabbitmq:image: rabbitmq:3.7.15-management
     container_name: rabbitmq
      restart: always
      volumes:-/data/rabbitmq/data: /var/lib/rabbitmq
          -/data/rabbitmq/log: /var/log/rabbitmq
      port:- 5672:5672- 15672:15672

docker-compose up -d

监控rabbitmq

docker安装exporter

docker直接运行

docker run -d-p9419:9419  --name rabbitmq_exporter -eRABBIT_URL=http://localhost:15672 -eRABBIT_USER=guest -eRABBIT_PASSWORD=guest kbudde/rabbitmq_exporter

docker-compose方式

cat>docker-compose.yaml <<FOF
version: '3.3'
services:
 rabbitmq_exproter:
     image: kbudde/rabbitmq-exporter
     container_name: rabbitmq_exporter
     restart: always
     environment:
         RABBIT_URL: "localhost:15672"
         RABBIT_USER: "guest"
         REDIS_PASSWORD: "guest"
         PUBLISH_PORT: "9419"
         OUTPUT_FORMAT: "JSON"
     port:
      - "9419:9419"
EOF

启动

docker-compose up -d

检查

查看正在运行的容器
dockerps

或者:

查看redis_exporter容器的运行日志
docker logs -f rabbitmq_exporter

参数解释

Environment variable值描述RABBIT_URLlocalhost:15672rabbitmq管理插件的url(必须以http(2)开头)RABBIT_USERguestrabbitmq管理插件的用户名REDIS_PASSWORDguestrabbitmq管理插件的密码OUTPUT_FORMATJSON输出格式PUBLISH_PORT9419运行端口(监听端口)

metrics地址

安装好Exporter后会暴露一个/metrics结尾的服务

名称地址rabbitmq_exporterhttp://localhost:9419/metrics

Prometheus配置

配置Prometheus去采集(拉取)nginx_exporter的监控样本数据

cd /data/docker-prometheus

# 在scrapc_configs(搜刮配置):下面增加如下配置:cat>prometheus/prometheus.yml <<FOF
 - job_name: 'rabbitmq_exporter'
   static_configs:
   - targets: ['localhost:9419']
        labels:
            instance: test服务器 
EOF

重新加载配置

curl-x POST http://localhost:9090/-/reload

检查

常用的rabbitmq监控指标

rabbitmq_queue_messages_unacknowledged_global            队列中有未确认的消息总数(未被消费的消息)
rabbitmq_node_disk_free_limit                            使用磁盘大小
rabbitmq_node_disk_free                                    磁盘总大小
    
rabbitmq_node_mem_used                                    使用内存大小
rabbitmq_node_mem_limit                                    内存总大小

rabbitmq_sockets_used                                     使用的sockets的数量        
rabbitmq_sockets_available                                 可用的sockets总数
    
rabbitmq_fd_used                                        使用文件描述符的数量
rabbitmq_fd_available                                    可用的文件描述符总数

触发器配置

由于之前的触发器是全部写在了一个yml里面就是alert.yam,这样随着后面配置的触发器越来越多最终会变得难以维护.这里我们让它去读rules目录下所有的yml文件即可

Prometheus配置

rule_files:
    - "alert.yml"
    - "rules/*.yml"

配置rabbitmq触发器

因为是单机所以未配置集群的触发器

cat>prometheus/rules/rabbitmq.yml <<FOF
groups:
- name: Rabbitmq
    rules:
    - alert: RabbitMQDown
      expr: rabbitmq_up !=1
      labels:
        severity: High
      annotations:
          summary: "Rabbitmq Down,实例:{{$labels.instance }}"
          description: "Rabbitmq_exporter连不上RabbitMQ!"
    - alert: Rabbitmq有未确认消息
      expr: rabbitmq_queue_messages_unacknowledged_global >0
      for: 1m
      labels:
        severity: critical
      annotations:
          summary: "RabbitMQ有未确认消息,实例:{{$labels.instance }}"
          description: "Rabbitmq未确认消息>0当前值为:{{ $value }}"
    - alert: RabbitMQ可用磁盘空间不足告警
      expr: rabbitmq_node_disk_free_alarm !=0
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "RabbitMQ可用磁盘空间不足,实例:{{$labels.instance }}"
          description: "RabbitMQ可用磁盘空间不足,请检查"
    - alert: RabbitMQ可用内存不足告警
      expr: rabbitmq_sockets_used / rabbitmq_sockets_available * 100>60
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "RabbitMQ可用内存不足,实例:{{$labels.instance }}"
          description: "RabbitMQ可用内存不足,请检查"
    - alert: RabbitMQ_socket连接数使用过高告警
      expr: rabbitmq_sockets_used / rabbitmq_sockets_available * 100>60
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "RabbitMQ_socket使用过高,实例:{{$labels.instance }}"
          description: "RabbitMQ_socket使用>60%当前值为::{{ $value }}"
    - alert: RabbitMQ文件描述符使用过高告警
      expr: rabbitmq_fd_used / rabbitmq_fd_available * 100>60
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "RaiibitMQ文件描述符使用过高,实例:{{$labels.instance }}"
          description: "RaiibitMQ文件描述符使用>60%,当前值为:{{ $value }}"
          
EOF

一定记住这里需要仔细校对yaml语法,最好是能去在线验证yaml语法的网站上看看.yaml语法还是比较严格的一点缩进错误都不能有

检查配置

dockerexec-it prometheus promtool check config /etc/prometheus/prometheus.yml

重新加载配置

curl-x POST http://localhost:9090/-/reload

检查

http://localhost:9090/alerts?search=

或:

http://localhost:9090/rules

dashboard

grafana展示Prometheus从redis_exporter收集到的数据

id :4279