prometheus通过blackbox-exporter监控web站点证书

1 概述

线上站点普遍是https，因此监控https web站点的证书的过期时间，是一个基础性需求。例如，证书过期会导致tls握手失败，进而导致用户无法正常访问web站点。
blackbox-expoter是一个web服务，它暴露了一个接口，访问这个接口能使得它去访问目标站点，并向客户端响应相关的web站点指标信息。prometheus和black-expoter结合使用，可以监控https web站点的响应时间、证书过期时间等。

2 blackbox-expoter

2.1 指标接口

格式：
GET /probe?module=模块名&target=<网址>
例子：
GET /probe?module=http_get_2xx&target=https://www.baidu.com

2.2 部署

blackbox-exporter的配置中定义了多种模块，例如ping，http_get_2xx等，模块名称可以自行定义。

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  
---

apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    k8s-app: blackbox-exporter
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 9115
    targetPort: 9115
  selector:
    k8s-app: blackbox-exporter
    
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    k8s-app: blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: blackbox-exporter
  template:
    metadata:
      labels:
        k8s-app: blackbox-exporter
    spec:
      containers:
      - name: blackbox-exporter
        image: prom/blackbox-exporter:latest
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --web.listen-address=:9115
        - --log.level=info
        ports:
        - name: http
          containerPort: 9115
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    app: blackbox-exporter
data:
  blackbox.yml: |-
    modules:
      ## ----------- TCP 检测模块配置 -----------
      tcp_connect:
        prober: tcp
        timeout: 5s
      ## ----------- ICMP 检测配置 -----------
      ping:
        prober: icmp
        timeout: 5s
        icmp:
          preferred_ip_protocol: "ip4"
      ## ----------- HTTP GET 2xx 检测模块配置 -----------
      http_get_2xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [200]           # 验证的HTTP状态码,默认为2xx
          no_follow_redirects: false          # 是否不跟随重定向
      ## ----------- HTTP GET 3xx 检测模块配置 -----------
      http_get_3xx:  
        prober: http
        timeout: 10s
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1","HTTP/2"]
          valid_status_codes: [301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xx
          no_follow_redirects: false                     # 是否不跟随重定向
      ## ----------- HTTP POST 监测模块 -----------
      http_post_2xx: 
        prober: http
        timeout: 10s
        http:
          method: POST
          preferred_ip_protocol: "ip4"
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          #headers:                             # HTTP头设置
          #  Content-Type: application/json
          #body: '{}'                           # 请求体设置

3 部署prometheus

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-app
  namespace: monitoring

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-app
  name: prometheus-app
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-app
  template:
    metadata:
      labels:
        app: prometheus-app
      name: prometheus-app
    spec:
      containers:
      - args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.retention=7d
        - --web.enable-lifecycle
        - --log.level=debug
        image: prom/prometheus:v2.31.0
        imagePullPolicy: IfNotPresent
        name: prometheus
        ports:
        - containerPort: 9090
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/prometheus
          name: config-volume
        - mountPath: /etc/prometheus/etc.d
          name: blackbox-web-target
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      serviceAccount: prometheus-app
      serviceAccountName: prometheus-app
      volumes:
      - configMap:
          name: prometheus-app
        name: config-volume
      - configMap:
          name: blackbox-web-target
        name: blackbox-web-target

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-app
    name: prometheus-app
  name: prometheus-app
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus-app
  sessionAffinity: None
  type: ClusterIP

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus-app
  namespace: monitoring

---
apiVersion: v1
data:
  prometheus.yml: |-
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: blackbox
      metrics_path: /probe
      params:
        module: [http_get_2xx]            #  会变成http的参数：module=http_get_2xx
      file_sd_configs:         
      - files: 
        - '/etc/prometheus/etc.d/web.yml'           # 被监控的目标站点是写在此文件
        refresh_interval: 30s                         # 30秒热更新一次，不必重启prometheus
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target            # 会变成http的参数：target=目标url
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
kind: ConfigMap
metadata:
  name: prometheus-app
  namespace: monitoring

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-web-target
  namespace: monitoring
  labels:
    app: blackbox-exporter
data:
  web.yml: |-
    ---
    - targets:
      - https://www.baidu.com         # 被监控的站点
      labels:
        env: prod
        app: baidu-web
        project: baidu
        desc: desc for baidu web
    - targets:
      - https://blog.csdn.net           # 被监控的站点
      labels:
        env: prod
        app: csdn-web
        project: csdn
        desc: desc for csdn

4 promethues界面效果

指标probe_ssl_earliest_cert_expiry表示证书的过期时间的时间戳，那么以下公式表示多少秒后证书过期：

probe_ssl_earliest_cert_expiry - time()  

5 grafana

5.1 部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:        
        - name: grafana
          image: grafana/grafana
          resources:
            limits:
              memory: "128Mi"
              cpu: "50m"
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            tcpSocket:
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 10
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
  - protocol: TCP
    port: 3000

5.2 配置数据源

添加prometheus数据源，prometheus实例在kubernetes中的service名称为prometheus-app，因此使用http://prometheus-app:9090作为地址即可。

5.3 导入模板

使用编号为13230的grafana模板。

6 小结

prometheus和blackbox-exporter一起协同监控web站点，blackbox-exporter作为一个中间层解耦prometheus和目标web站点，blackbox-exporter是真正去获取目标web站点证书并暴露metrics的服务，prometheus只需要抓取blackbox-exporter暴露的指标即可。