Kubernetes 部署 RabbitMQ

文章目录

• Kubernetes

# Kubernetes
https://iothub.org.cn/docs/kubernetes/
https://iothub.org.cn/docs/kubernetes/middleware/rabbitmq/

一、概述

1.Operator

# awesome-operators
https://github.com/operator-framework/awesome-operators
# 安装# RabbitMQ #1 (Official)  rabbitmq/cluster-operator
https://github.com/rabbitmq/cluster-operator

App NameGithubDescriptionRabbitMQ #1 (Official)rabbitmq/cluster-operatorRabbitMQ cluster operator for Kubernetes.RabbitMQ #2skylt/rabbitmq-operatorRabbitMQ operator for Kubernetes.

2.Helm

Bitnami

https://bitnami.com/
https://github.com/bitnami
https://bitnami.com/stacks

[root@k8s-master redis-cluster-operator]# helm search repo rabbitmq
NAME                                 CHART VERSION    APP VERSION    DESCRIPTION                                 
bitnami/rabbitmq                     8.23.0           3.9.7          Open source message broker software that implem...
bitnami/rabbitmq-cluster-operator    0.1.6            1.9.0          The RabbitMQ Cluster Kubernetes Operator automa...
......

二、基础

1.Operator

# 安装# RabbitMQ #1 (Official)  rabbitmq/cluster-operator
https://github.com/rabbitmq/cluster-operator
#RabbitMQ Cluster Kubernetes Operator Quickstart
https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html
# Using RabbitMQ Cluster Kubernetes Operator
https://www.rabbitmq.com/kubernetes/operator/using-operator.html

# 参考# Operator overview
https://www.rabbitmq.com/kubernetes/operator/operator-overview.html
# Deploying an operator
https://www.rabbitmq.com/kubernetes/operator/install-operator.html
# Deploying a RabbitMQ cluster
https://www.rabbitmq.com/kubernetes/operator/using-operator.html
# Monitoring the cluster
https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html
# Troubleshooting operator deployments
https://www.rabbitmq.com/kubernetes/operator/troubleshooting-operator.html

下载rabbitmq/cluster-operator

# git  git clone https://github.com/rabbitmq/cluster-operator.git[root@k8s-master rabbitmq]# git clone https://github.com/rabbitmq/cluster-operator.git
Cloning into 'cluster-operator'...
remote: Enumerating objects: 11892, done.
remote: Counting objects: 100% (2279/2279), done.
remote: Compressing objects: 100% (976/976), done.
remote: Total 11892(delta 1438), reused 1976(delta 1240), pack-reused 9613
Receiving objects: 100% (11892/11892), 11.95 MiB |1.66 MiB/s, done.
Resolving deltas: 100% (7740/7740), done.

1.1.Quickstart（单实例）

#RabbitMQ Cluster Kubernetes Operator Quickstart
https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html
Quickstart Steps
This guide goes through the following steps:
Install the RabbitMQ Cluster Operator
Deploy a RabbitMQ Cluster using the Operator
View RabbitMQ Logs
Access the RabbitMQ Management UI
Attach a Workload to the Cluster
Next Steps

1.创建Operator

#kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml[root@k8s-master docs]# kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created
# 查看# 会创建一个新的名称空间：rabbitmq-system[root@k8s-master docs]# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-5c4wk   1/1     Running   0          81s
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     11           81s
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   111       81s

2.使用cluster-operator创建RabbitMQ

简单的yaml文件下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/hello-world
yaml文件名：rabbitmq.yaml
这是最简单的RabbitmqCluster定义。唯一显式指定的属性是集群的名称。其他一切都将根据集群运营商的默认值进行配置。

examples文件夹还有许多其他引用，比如用TLS、mTLS创建RabbitMQ集群，用生产默认值设置集群，添加社区插件等等。下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/

# rabbitmq.yamlapiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:name: hello-world

# kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml[root@k8s-master docs]# kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml
rabbitmqcluster.rabbitmq.com/hello-world created
[root@k8s-master docs]# kubectl get all
NAME                       READY   STATUS    RESTARTS   AGE
pod/hello-world-server-0   0/1     Pending   0          74s
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
service/hello-world         ClusterIP   10.100.123.79   <none>15672/TCP,15692/TCP,5672/TCP   74s
service/hello-world-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             74s
NAME                                  READY   AGE
statefulset.apps/hello-world-server   0/1     74s
NAME                                       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/hello-world   False              Unknown            74s
[root@k8s-master rabbitmq]# kubectl get pvc
NAME                               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistence-hello-world-server-0   Pending                                                     16m
# 查找Pending愿意，没有存储[root@k8s-master docs]# kubectl describe pod hello-world-server-0
Name:           hello-world-server-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/component=rabbitmq
                app.kubernetes.io/name=hello-world
                app.kubernetes.io/part-of=rabbitmq
                controller-revision-hash=hello-world-server-6669f4bc8f
                statefulset.kubernetes.io/pod-name=hello-world-server-0
Annotations:    prometheus.io/port: 15692
                prometheus.io/scrape: true
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/hello-world-server
Init Containers:
  setup-container:
    Image:      rabbitmq:3.8.21-management
    Port:       <none>
    Host Port:  <none>
    Command:
      sh-ccp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie &&chmod600 /var/lib/rabbitmq/.erlang.cookie ;cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ;echo'[default]'> /var/lib/rabbitmq/.rabbitmqadmin.conf &&sed-e's/default_user/username/'-e's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf &&chmod600 /var/lib/rabbitmq/.rabbitmqadmin.conf
    Limits:
      cpu:     100m
      memory:  500Mi
    Requests:
      cpu:        100m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /operator from rabbitmq-plugins (rw)
      /tmp/default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
      /tmp/rabbitmq-plugins/ from plugins-conf (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from hello-world-server-token-xvxcp (ro)
Containers:
  rabbitmq:
    Image:       rabbitmq:3.8.21-management
    Ports:       4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Readiness:  tcp-socket :amqp delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAME:                    hello-world-server-0 (v1:metadata.name)
      MY_POD_NAMESPACE:               default (v1:metadata.namespace)
      K8S_SERVICE_NAME:               hello-world-nodes
      RABBITMQ_ENABLED_PLUGINS_FILE:  /operator/enabled_plugins
      RABBITMQ_USE_LONGNAME:          true
      RABBITMQ_NODENAME:              rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
      K8S_HOSTNAME_SUFFIX:            .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
    Mounts:
      /etc/pod-info/ from pod-info (rw)
      /etc/rabbitmq/conf.d/10-operatorDefaults.conf from rabbitmq-confd (rw,path="operatorDefaults.conf")
      /etc/rabbitmq/conf.d/11-default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf from rabbitmq-confd (rw,path="userDefinedConfiguration.conf")
      /operator from rabbitmq-plugins (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from hello-world-server-token-xvxcp (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  persistence:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  persistence-hello-world-server-0
    ReadOnly:   false
  plugins-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      hello-world-plugins-conf
    Optional:  false
  rabbitmq-confd:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       hello-world-server-conf
    ConfigMapOptional:   <nil>
    SecretName:          hello-world-default-user
    SecretOptionalName:  <nil>
  rabbitmq-erlang-cookie:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  erlang-cookie-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hello-world-erlang-cookie
    Optional:    false
  rabbitmq-plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
  hello-world-server-token-xvxcp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hello-world-server-token-xvxcp
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  8m21s  default-scheduler  0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  8m21s  default-scheduler  0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.

# 手动创建PVC# persistence-hello-world-server-0# persistence-hello-world-server-0.yamlapiVersion: v1
kind: PersistentVolumeClaim
metadata:name: persistence-hello-world-server-0namespace: default
spec:accessModes:- ReadWriteOnce
  storageClassName: rook-ceph-block
  resources:requests:storage: 12Gi
      
[root@k8s-master rabbitmq]# kubectl apply -f persistence-hello-world-server-0.yaml 
persistentvolumeclaim/persistence-hello-world-server-0 created
[root@k8s-master rabbitmq]# kubectl get pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-hello-world-server-0   Bound    pvc-91f82092-4b29-42ab-87b4-2cf3f9099089   12Gi       RWO            rook-ceph-block   3s

# 服务正常[root@k8s-master rabbitmq]# kubectl get all
NAME                       READY   STATUS    RESTARTS   AGE
pod/hello-world-server-0   1/1     Running   0          20m
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
service/hello-world         ClusterIP   10.100.123.79   <none>15672/TCP,15692/TCP,5672/TCP   20m
service/hello-world-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             20m
NAME                                  READY   AGE
statefulset.apps/hello-world-server   1/1     20m
NAME                                       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/hello-world   True               True               20m
[root@k8s-master rabbitmq]# kubectl get pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-hello-world-server-0   Bound    pvc-91f82092-4b29-42ab-87b4-2cf3f9099089   12Gi       RWO            rook-ceph-block   2m27s

1.2.获取用户名和密码

获取用户名和密码

# 获取获取用户名和密码username="$(kubectl get secret hello-world-default-user -ojsonpath='{.data.username}'| base64 --decode)"echo"username: $username"password="$(kubectl get secret hello-world-default-user -ojsonpath='{.data.password}'| base64 --decode)"echo"password: $password"[root@k8s-master rabbitmq]# username="$(kubectl get secret hello-world-default-user -o jsonpath='{.data.username}' | base64 --decode)"[root@k8s-master rabbitmq]# echo "username: $username"
username: default_user_HGTmPURsI5VhaVjmeEQ
[root@k8s-master rabbitmq]# password="$(kubectl get secret hello-world-default-user -o jsonpath='{.data.password}' | base64 --decode)"[root@k8s-master rabbitmq]# echo "password: $password"
password: TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX
# 登录信息：
username: default_user_HGTmPURsI5VhaVjmeEQ
password: TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX
kubectl port-forward "service/hello-world"15672

# 系统创建service# hello-world[root@k8s-master rabbitmq]# kubectl get svc
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
hello-world         ClusterIP   10.100.123.79   <none>15672/TCP,15692/TCP,5672/TCP   33m
hello-world-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             33m
[root@k8s-master rabbitmq]# kubectl port-forward "service/hello-world" 15672
Forwarding from 127.0.0.1:15672 ->15672
Forwarding from [::1]:15672 ->15672curl -u$username:$password localhost:15672/api/overview
{"management_version":"3.8.9","rates_mode":"basic", ...}curl-udefault_user_HGTmPURsI5VhaVjmeEQ:TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX10.100.123.79:15672/api/overview
# 访问[root@k8s-master rabbitmq]# curl -udefault_user_HGTmPURsI5VhaVjmeEQ:TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX 10.100.123.79:15672/api/overview{"management_version":"3.8.21","rates_mode":"basic","sample_retention_policies":{"global":[600,3600,28800,86400],"basic":[600,3600],"detailed":[600]},"exchange_types":[{"name":"direct","description":"AMQP direct exchange, as per the AMQP specification","enabled":true},{"name":"fanout","description":"AMQP fanout exchange, as per the AMQP specification","enabled":true},{"name":"headers","description":"AMQP headers exchange, as per the AMQP specification","enabled":true},{"name":"topic","description":"AMQP topic exchange, as per the AMQP specification","enabled":true}],"product_version":"3.8.21","product_name":"RabbitMQ","rabbitmq_version":"3.8.21","cluster_name":"hello-world","erlang_version":"24.0.5","erlang_full_version":"Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:4:2] [ds:4:2:10] [async-threads:1] [jit]","disable_stats":false,"enable_queue_totals":false,"message_stats":{},"churn_rates":{"channel_closed":0,"channel_closed_details":{"rate":0.0},"channel_created":0,"channel_created_details":{"rate":0.0},"connection_closed":312,"connection_closed_details":{"rate":0.2},"connection_created":0,"connection_created_details":{"rate":0.0},"queue_created":0,"queue_created_details":{"rate":0.0},"queue_declared":0,"queue_declared_details":{"rate":0.0},"queue_deleted":0,"queue_deleted_details":{"rate":0.0}},"queue_totals":{},"object_totals":{"channels":0,"connections":0,"consumers":0,"exchanges":7,"queues":0},"statistics_db_event_queue":0,"node":"rabbit@hello-world-server-0.hello-world-nodes.default","listeners":[{"node":"rabbit@hello-world-server-0.hello-world-nodes.default","protocol":"amqp","ip_address":"::","port":5672,"socket_opts":{"backlog":128,"nodelay":true,"linger":[true,0],"exit_on_close":false}},{"node":"rabbit@hello-world-server-0.hello-world-nodes.default","protocol":"clustering","ip_address":"::","port":25672,"socket_opts":[]},{"node":"rabbit@hello-world-server-0.hello-world-nodes.default","protocol":"http","ip_address":"::","port":15672,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15672}},{"node":"rabbit@hello-world-server-0.hello-world-nodes.default","protocol":"http/prometheus","ip_address":"::","port":15692,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15692,"protocol":"http/prometheus"}}],"contexts":[{"ssl_opts":[],"node":"rabbit@hello-world-server-0.hello-world-nodes.default","description":"RabbitMQ Management","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15672"},{"ssl_opts":[],"node":"rabbit@hello-world-server-0.hello-world-nodes.default","description":"RabbitMQ Prometheus","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15692","protocol":"'http/prometheus'"}]}

1.3.访问RabbitMQ管理UI

rabbitmq-k8s-svc.yaml

# rabbitmq-k8s-svc.yamlapiVersion: v1
kind: Service
metadata:namespace: default
  name: rabbitmq-k8s-svc
  labels:statefulset.kubernetes.io/pod-name: hello-world-server-0spec:type: NodePort
  ports:-port:15672name: rabbitmq-http
      targetPort:15672nodePort:30672-port:5672name: rabbitmq-tcp
      targetPort:5672nodePort:30673selector:statefulset.kubernetes.io/pod-name: hello-world-server-0

[root@k8s-master rabbitmq]# kubectl apply -f rabbitmq-svc.yaml 
service/rabbitmq-svc created
[root@k8s-master rabbitmq]# kubectl get svc
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
hello-world         ClusterIP   10.100.123.79   <none>15672/TCP,15692/TCP,5672/TCP   141m
hello-world-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             141m
rabbitmq-k8s-svc    NodePort    10.96.247.212   <none>15672:30672/TCP                6s
curl-udefault_user_HGTmPURsI5VhaVjmeEQ:TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX10.96.247.212:15672/api/overview

# 地址
http://172.51.216.81:30672
username: default_user_HGTmPURsI5VhaVjmeEQ
password: TawohNwWLpQ0D7xmpElx9bUsL_PuIhhX

1.4.删除Rabbitmq

# 删除Rabbitmq# 根据创建选择
kubectl delete -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml
# 删除operator
kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

# 查看[root@k8s-master rabbitmq]# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-5c4wk   1/1     Running   1          19h
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     11           19h
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   111       19h
[root@k8s-master rabbitmq]# kubectl get all
NAME                       READY   STATUS    RESTARTS   AGE
pod/hello-world-server-0   1/1     Running   1          19h
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
service/hello-world         ClusterIP   10.100.123.79    <none>15672/TCP,15692/TCP,5672/TCP     19h
service/hello-world-nodes   ClusterIP   None             <none>4369/TCP,25672/TCP               19h
service/rabbitmq-k8s-svc    NodePort    10.106.168.227   <none>15672:30672/TCP,5672:30673/TCP   29m
NAME                                  READY   AGE
statefulset.apps/hello-world-server   1/1     19h
NAME                                       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/hello-world   True               True               19h
# 删除# 删除svc[root@k8s-master rabbitmq]# kubectl delete -f rabbitmq-k8s-svc.yaml service"rabbitmq-k8s-svc" deleted
# 删除rabbitmq[root@k8s-master rabbitmq]# kubectl delete -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml
rabbitmqcluster.rabbitmq.com "hello-world" deleted
[root@k8s-master rabbitmq]# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>443/TCP   105d
# 删除operator[root@k8s-master rabbitmq]# kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
namespace "rabbitmq-system" deleted
customresourcedefinition.apiextensions.k8s.io "rabbitmqclusters.rabbitmq.com" deleted
serviceaccount "rabbitmq-cluster-operator" deleted
role.rbac.authorization.k8s.io "rabbitmq-cluster-leader-election-role" deleted
clusterrole.rbac.authorization.k8s.io "rabbitmq-cluster-operator-role" deleted
rolebinding.rbac.authorization.k8s.io "rabbitmq-cluster-leader-election-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "rabbitmq-cluster-operator-rolebinding" deleted
deployment.apps "rabbitmq-cluster-operator" deleted
[root@k8s-master rabbitmq]# kubectl get all -n rabbitmq-system
No resources found in rabbitmq-system namespace.
# 删除PVC[root@k8s-master rabbitmq]# kubectl delete -f persistence-hello-world-server-0.yaml 
persistentvolumeclaim "persistence-hello-world-server-0" deleted
[root@k8s-master rabbitmq]# kubectl get pvc
No resources found in default namespace.

1.5.安装方式选择

单实例安装方式

yaml文件下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/hello-world
yaml文件名：rabbitmq.yaml

这是最简单的RabbitmqCluster定义。唯一显式指定的属性是集群的名称。其他一切都将根据集群运营商的默认值进行配置。

examples文件夹还有许多其他引用，比如用TLS、mTLS创建RabbitMQ集群，用生产默认值设置集群，添加社区插件等等。

下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/

多种安装方式

https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples

[root@k8s-master examples]# pwd
/k8s/middleware/rabbitmq/cluster-operator/docs/examples
[root@k8s-master examples]# ll
total 4
drwxr-xr-x 2 root root   59 Nov 3011:52 additionalPorts
drwxr-xr-x 2 root root   59 Nov 3011:52 community-plugins
drwxr-xr-x 2 root root   60 Nov 3011:52 custom-configuration
drwxr-xr-x 2 root root   59 Nov 3011:52 default-security-context
drwxr-xr-x 2 root root  177 Nov 3011:52 federation-over-tls
drwxr-xr-x 2 root root   60 Nov 3011:52 hello-world
drwxr-xr-x 2 root root   99 Nov 3011:52 import-definitions
drwxr-xr-x 2 root root   59 Nov 3011:52 json-log
drwxr-xr-x 2 root root   60 Nov 3011:52 mtls
drwxr-xr-x 2 root root  161 Nov 3011:52 mtls-inter-node
drwxr-xr-x 2 root root   59 Nov 3011:52 multiple-disks
drwxr-xr-x 2 root root  167 Nov 3011:52 network-policies
drwxr-xr-x 2 root root   60 Nov 3011:52 plugins
drwxr-xr-x 2 root root   60 Nov 3011:52 pod-anti-affinity
drwxr-xr-x 2 root root  114 Nov 3011:52 production-ready
-rw-r--r-- 1 root root 2372 Nov 3011:52 README.md
drwxr-xr-x 2 root root   60 Nov 3011:52 resource-limits
drwxr-xr-x 2 root root   59 Nov 3011:52 set-login-password-username
drwxr-xr-x 2 root root   59 Nov 3011:52 sidecar
drwxr-xr-x 2 root root   99 Nov 3011:52 tls
drwxr-xr-x 2 root root   60 Nov 3011:52 tolerations
drwxr-xr-x 2 root root   75 Nov 3011:52 vault-default-user
drwxr-xr-x 2 root root   75 Nov 3011:52 vault-tls

# 单实例部署
drwxr-xr-x 2 root root   60 Nov 3011:52 hello-world
# 集群部署
drwxr-xr-x 2 root root  114 Nov 3011:52 production-ready

三、实践

1.部署说明

集群部署：

https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples

https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/production-ready

安装说明

https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/production-ready/README.md

[root@k8s-master examples]# cd production-ready/[root@k8s-master production-ready]# ll
total 16
-rw-r--r-- 1 root root  199 Nov 3011:52 pod-disruption-budget.yaml
-rw-r--r-- 1 root root 1184 Nov 3011:52 rabbitmq.yaml
-rw-r--r-- 1 root root 3134 Nov 3011:52 README.md
-rw-r--r-- 1 root root  409 Nov 3011:52 ssd-gke.yaml
# 部署
kubectl apply -f rabbitmq.yaml
kubectl apply -f pod-disruption-budget.yaml

配置文件

# rabbitmq.yaml[root@k8s-master production-ready]# vim rabbitmq.yaml apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:name: production-ready
spec:replicas:3resources:requests:cpu:4memory: 10Gi
    limits:cpu:4memory: 10Gi
  rabbitmq:additionalConfig:|
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000persistence:storageClassName: ssd
    storage:"500Gi"affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key: app.kubernetes.io/name
              operator: In
              values:- production-ready
        topologyKey: kubernetes.io/hostname
  override:statefulSet:spec:template:spec:containers:[]topologySpreadConstraints:-maxSkew:1topologyKey:"topology.kubernetes.io/zone"whenUnsatisfiable: DoNotSchedule
              labelSelector:matchLabels:app.kubernetes.io/name: production-ready

# pod-disruption-budget.yaml [root@k8s-master production-ready]# vim pod-disruption-budget.yaml apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:name: production-ready-rabbitmq
spec:maxUnavailable:1selector:matchLabels:app.kubernetes.io/name: production-ready

2.部署RabbitMQ集群

1.下载rabbitmq/cluster-operator

# git  git clone https://github.com/rabbitmq/cluster-operator.git[root@k8s-master rabbitmq]# git clone https://github.com/rabbitmq/cluster-operator.git
Cloning into 'cluster-operator'...
remote: Enumerating objects: 11892, done.
remote: Counting objects: 100% (2279/2279), done.
remote: Compressing objects: 100% (976/976), done.
remote: Total 11892(delta 1438), reused 1976(delta 1240), pack-reused 9613
Receiving objects: 100% (11892/11892), 11.95 MiB |1.66 MiB/s, done.
Resolving deltas: 100% (7740/7740), done.

2.创建Operator

# kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml# 创建[root@k8s-master rabbitmq]# kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created
[root@k8s-master rabbitmq]# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-s95gq   1/1     Running   0          33s
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     11           33s
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   111       33s

3.修改配置

# rabbitmq.yaml# cpu、memory# storageClassName: rook-ceph-block[root@k8s-master production-ready]# vim rabbitmq.yaml apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:name: production-ready
spec:replicas:3resources:requests:cpu:1memory: 2Gi
    limits:cpu:1memory: 2Gi
  rabbitmq:additionalConfig:|
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000persistence:storageClassName: rook-ceph-block
    storage:"12Gi"

4.使用cluster-operator创建RabbitMQ集群

[root@k8s-master production-ready]# pwd
/k8s/middleware/rabbitmq/cluster-operator/docs/examples/production-ready
[root@k8s-master production-ready]# ll
total 16
-rw-r--r-- 1 root root  199 Nov 3011:52 pod-disruption-budget.yaml
-rw-r--r-- 1 root root 1193 Dec  110:28 rabbitmq.yaml
-rw-r--r-- 1 root root 3134 Nov 3011:52 README.md
-rw-r--r-- 1 root root  409 Nov 3011:52 ssd-gke.yaml
# 创建[root@k8s-master production-ready]# kubectl apply -f rabbitmq.yaml
rabbitmqcluster.rabbitmq.com/production-ready created
[root@k8s-master production-ready]# [root@k8s-master production-ready]# kubectl apply -f pod-disruption-budget.yaml 
poddisruptionbudget.policy/production-ready-rabbitmq created
# 查看[root@k8s-master production-ready]# kubectl get all -owide
NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
pod/production-ready-server-0   1/1     Running   0          50s   10.244.36.117    k8s-node1   <none><none>
pod/production-ready-server-1   1/1     Running   0          50s   10.244.169.164   k8s-node2   <none><none>
pod/production-ready-server-2   1/1     Running   0          50s   10.244.107.251   k8s-node3   <none><none>
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE    SELECTOR
service/kubernetes               ClusterIP   10.96.0.1       <none>443/TCP                        105d   <none>
service/production-ready         ClusterIP   10.111.187.65   <none>5672/TCP,15672/TCP,15692/TCP   50s    app.kubernetes.io/name=production-ready
service/production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             50s    app.kubernetes.io/name=production-ready
NAME                                       READY   AGE   CONTAINERS   IMAGES
statefulset.apps/production-ready-server   3/3     50s   rabbitmq     rabbitmq:3.8.21-management
NAME                                            ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/production-ready   True               True               50s
[root@k8s-master production-ready]# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-s95gq   1/1     Running   0          105m
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     11           105m
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   111       105m
# PVC[root@k8s-master production-ready]# kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-production-ready-server-0   Bound    pvc-71d4c7b8-554c-4f9e-856e-325e80ed88f2   12Gi       RWO            rook-ceph-block   6m10s
persistence-production-ready-server-1   Bound    pvc-6be2c9b3-a0b0-43ae-af1f-74ae92bce5ce   12Gi       RWO            rook-ceph-block   6m10s
persistence-production-ready-server-2   Bound    pvc-2243a65e-f38e-4164-a2f1-ec0a65be486d   12Gi       RWO            rook-ceph-block   6m10s

3.获取用户名和密码

获取用户名和密码

# user
kubectl -n NAMESPACE get secret INSTANCE-default-user -ojsonpath="{.data.username}"| base64 --decode# pass
kubectl -n NAMESPACE get secret INSTANCE-default-user -ojsonpath="{.data.password}"| base64 --decode# 获取获取用户名和密码  production-readyusername="$(kubectl get secret production-ready-default-user -ojsonpath='{.data.username}'| base64 --decode)"echo"username: $username"password="$(kubectl get secret production-ready-default-user -ojsonpath='{.data.password}'| base64 --decode)"echo"password: $password"[root@k8s-master production-ready]# username="$(kubectl get secret production-ready-default-user -o jsonpath='{.data.username}' | base64 --decode)"[root@k8s-master production-ready]# echo "username: $username"
username: default_user_vmOg0aiF8VarwVhb0sE
[root@k8s-master production-ready]# password="$(kubectl get secret production-ready-default-user -o jsonpath='{.data.password}' | base64 --decode)"[root@k8s-master production-ready]# echo "password: $password"
password: rKr9N2v-Cxrec78zo1LUnigjyOla_YLv
# 登录信息：
username: default_user_vmOg0aiF8VarwVhb0sE
password: rKr9N2v-Cxrec78zo1LUnigjyOla_YLv
kubectl port-forward "service/production-ready"15672

# 系统创建service# production-ready[root@k8s-master production-ready]# kubectl get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
production-ready         ClusterIP   10.111.187.65   <none>5672/TCP,15672/TCP,15692/TCP   6m48s
production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP             6m48s
[root@k8s-master production-ready]# kubectl port-forward "service/production-ready" 15672
Forwarding from 127.0.0.1:15672 ->15672
Forwarding from [::1]:15672 ->15672curl -u$username:$password localhost:15672/api/overview
{"management_version":"3.8.9","rates_mode":"basic", ...}curl -udefault_user_vmOg0aiF8VarwVhb0sE:rKr9N2v-Cxrec78zo1LUnigjyOla_YLv 10.111.187.65:15672/api/overview
# 访问[root@k8s-master production-ready]# curl -udefault_user_vmOg0aiF8VarwVhb0sE:rKr9N2v-Cxrec78zo1LUnigjyOla_YLv 10.111.187.65:15672/api/overview{"management_version":"3.8.21","rates_mode":"basic","sample_retention_policies":{"global":[600,3600,28800,86400],"basic":[600,3600],"detailed":[600]},"exchange_types":[{"name":"direct","description":"AMQP direct exchange, as per the AMQP specification","enabled":true},{"name":"fanout","description":"AMQP fanout exchange, as per the AMQP specification","enabled":true},{"name":"headers","description":"AMQP headers exchange, as per the AMQP specification","enabled":true},{"name":"topic","description":"AMQP topic exchange, as per the AMQP specification","enabled":true}],"product_version":"3.8.21","product_name":"RabbitMQ","rabbitmq_version":"3.8.21","cluster_name":"production-ready","erlang_version":"24.0.5","erlang_full_version":"Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:4:1] [ds:4:1:10] [async-threads:1] [jit]","disable_stats":false,"enable_queue_totals":false,"message_stats":{},"churn_rates":{"channel_closed":0,"channel_closed_details":{"rate":0.0},"channel_created":0,"channel_created_details":{"rate":0.0},"connection_closed":153,"connection_closed_details":{"rate":0.2},"connection_created":0,"connection_created_details":{"rate":0.0},"queue_created":0,"queue_created_details":{"rate":0.0},"queue_declared":0,"queue_declared_details":{"rate":0.0},"queue_deleted":0,"queue_deleted_details":{"rate":0.0}},"queue_totals":{},"object_totals":{"channels":0,"connections":0,"consumers":0,"exchanges":7,"queues":0},"statistics_db_event_queue":0,"node":"rabbit@production-ready-server-1.production-ready-nodes.default","listeners":[{"node":"rabbit@production-ready-server-0.production-ready-nodes.default","protocol":"amqp","ip_address":"::","port":5672,"socket_opts":{"backlog":128,"nodelay":true,"linger":[true,0],"exit_on_close":false}},{"node":"rabbit@production-ready-server-1.production-ready-nodes.default","protocol":"amqp","ip_address":"::","port":5672,"socket_opts":{"backlog":128,"nodelay":true,"linger":[true,0],"exit_on_close":false}},{"node":"rabbit@production-ready-server-2.production-ready-nodes.default","protocol":"amqp","ip_address":"::","port":5672,"socket_opts":{"backlog":128,"nodelay":true,"linger":[true,0],"exit_on_close":false}},{"node":"rabbit@production-ready-server-0.production-ready-nodes.default","protocol":"clustering","ip_address":"::","port":25672,"socket_opts":[]},{"node":"rabbit@production-ready-server-1.production-ready-nodes.default","protocol":"clustering","ip_address":"::","port":25672,"socket_opts":[]},{"node":"rabbit@production-ready-server-2.production-ready-nodes.default","protocol":"clustering","ip_address":"::","port":25672,"socket_opts":[]},{"node":"rabbit@production-ready-server-0.production-ready-nodes.default","protocol":"http","ip_address":"::","port":15672,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15672}},{"node":"rabbit@production-ready-server-1.production-ready-nodes.default","protocol":"http","ip_address":"::","port":15672,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15672}},{"node":"rabbit@production-ready-server-2.production-ready-nodes.default","protocol":"http","ip_address":"::","port":15672,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15672}},{"node":"rabbit@production-ready-server-0.production-ready-nodes.default","protocol":"http/prometheus","ip_address":"::","port":15692,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15692,"protocol":"http/prometheus"}},{"node":"rabbit@production-ready-server-1.production-ready-nodes.default","protocol":"http/prometheus","ip_address":"::","port":15692,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15692,"protocol":"http/prometheus"}},{"node":"rabbit@production-ready-server-2.production-ready-nodes.default","protocol":"http/prometheus","ip_address":"::","port":15692,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15692,"protocol":"http/prometheus"}}],"contexts":[{"ssl_opts":[],"node":"rabbit@production-ready-server-0.production-ready-nodes.default","description":"RabbitMQ Management","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15672"},{"ssl_opts":[],"node":"rabbit@production-ready-server-1.production-ready-nodes.default","description":"RabbitMQ Management","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15672"},{"ssl_opts":[],"node":"rabbit@production-ready-server-2.production-ready-nodes.default","description":"RabbitMQ Management","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15672"},{"ssl_opts":[],"node":"rabbit@production-ready-server-0.production-ready-nodes.default","description":"RabbitMQ Prometheus","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15692","protocol":"'http/prometheus'"},{"ssl_opts":[],"node":"rabbit@production-ready-server-1.production-ready-nodes.default","description":"RabbitMQ Prometheus","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15692","protocol":"'http/prometheus'"},{"ssl_opts":[],"node":"rabbit@production-ready-server-2.production-ready-nodes.default","description":"RabbitMQ Prometheus","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15692","protocol":"'http/prometheus'"}]}[root@k8s-master production-ready]# 

4.访问RabbitMQ管理UI

rabbitmq-cluster-svc.yaml

# rabbitmq-cluster-svc.yamlapiVersion: v1
kind: Service
metadata:namespace: default
  name: rabbitmq-cluster-svc
  labels:app.kubernetes.io/name: production-ready
spec:type: NodePort
  ports:-port:15672name: rabbitmq-http
      targetPort:15672nodePort:30672-port:5672name: rabbitmq-tcp
      targetPort:5672nodePort:30673selector:app.kubernetes.io/name: production-ready

[root@k8s-master rabbitmq]# kubectl apply -f rabbitmq-svc.yaml 
service/rabbitmq-svc created
[root@k8s-master rabbitmq]# kubectl get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
production-ready         ClusterIP   10.111.187.65   <none>5672/TCP,15672/TCP,15692/TCP     10m
production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP               10m
rabbitmq-cluster-svc     NodePort    10.106.36.141   <none>15672:30672/TCP,5672:30673/TCP   5s
curl -udefault_user_vmOg0aiF8VarwVhb0sE:rKr9N2v-Cxrec78zo1LUnigjyOla_YLv 10.106.36.141:15672/api/overview

# 地址
http://172.51.216.81:30672
username: default_user_vmOg0aiF8VarwVhb0sE
password: rKr9N2v-Cxrec78zo1LUnigjyOla_YLv

5.扩容

修改配置

# rabbitmq.yaml# cpu、memory# storageClassName: rook-ceph-block[root@k8s-master production-ready]# vim rabbitmq.yaml apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:name: production-ready
spec:replicas:4resources:requests:cpu:1memory: 2Gi
    limits:cpu:1memory: 2Gi
  rabbitmq:additionalConfig:|
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000persistence:storageClassName: rook-ceph-block
    storage:"12Gi"

[root@k8s-master production-ready]# kubectl apply -f rabbitmq.yaml 
rabbitmqcluster.rabbitmq.com/production-ready configured
# 查看[root@k8s-master production-ready]# kubectl get all -owide
NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
pod/production-ready-server-0   1/1     Running   0          24m   10.244.36.117    k8s-node1   <none><none>
pod/production-ready-server-1   1/1     Running   0          24m   10.244.169.164   k8s-node2   <none><none>
pod/production-ready-server-2   1/1     Running   0          24m   10.244.107.251   k8s-node3   <none><none>
pod/production-ready-server-3   1/1     Running   0          45s   10.244.169.159   k8s-node2   <none><none>
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE    SELECTOR
service/kubernetes               ClusterIP   10.96.0.1       <none>443/TCP                          105d   <none>
service/production-ready         ClusterIP   10.111.187.65   <none>5672/TCP,15672/TCP,15692/TCP     24m    app.kubernetes.io/name=production-ready
service/production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP               24m    app.kubernetes.io/name=production-ready
service/rabbitmq-cluster-svc     NodePort    10.106.36.141   <none>15672:30672/TCP,5672:30673/TCP   14m    app.kubernetes.io/name=production-ready
NAME                                       READY   AGE   CONTAINERS   IMAGES
statefulset.apps/production-ready-server   4/4     24m   rabbitmq     rabbitmq:3.8.21-management
NAME                                            ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/production-ready   True               True               24m
[root@k8s-master production-ready]# kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-production-ready-server-0   Bound    pvc-71d4c7b8-554c-4f9e-856e-325e80ed88f2   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-1   Bound    pvc-6be2c9b3-a0b0-43ae-af1f-74ae92bce5ce   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-2   Bound    pvc-2243a65e-f38e-4164-a2f1-ec0a65be486d   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-3   Bound    pvc-26fca32f-d0d9-44af-8378-1a2aeefb1d94   12Gi       RWO            rook-ceph-block   3m21s

6.缩容

修改配置

# rabbitmq.yaml# cpu、memory# storageClassName: rook-ceph-block[root@k8s-master production-ready]# vim rabbitmq.yaml apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:name: production-ready
spec:replicas:3resources:requests:cpu:1memory: 2Gi
    limits:cpu:1memory: 2Gi
  rabbitmq:additionalConfig:|
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000persistence:storageClassName: rook-ceph-block
    storage:"12Gi"

[root@k8s-master production-ready]# kubectl apply -f rabbitmq.yaml 
rabbitmqcluster.rabbitmq.com/production-ready configured
# 查看[root@k8s-master production-ready]# kubectl get all -owide
NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
pod/production-ready-server-0   1/1     Running   0          24m   10.244.36.117    k8s-node1   <none><none>
pod/production-ready-server-1   1/1     Running   0          24m   10.244.169.164   k8s-node2   <none><none>
pod/production-ready-server-2   1/1     Running   0          24m   10.244.107.251   k8s-node3   <none><none>
pod/production-ready-server-3   1/1     Running   0          45s   10.244.169.159   k8s-node2   <none><none>
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE    SELECTOR
service/kubernetes               ClusterIP   10.96.0.1       <none>443/TCP                          105d   <none>
service/production-ready         ClusterIP   10.111.187.65   <none>5672/TCP,15672/TCP,15692/TCP     24m    app.kubernetes.io/name=production-ready
service/production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP               24m    app.kubernetes.io/name=production-ready
service/rabbitmq-cluster-svc     NodePort    10.106.36.141   <none>15672:30672/TCP,5672:30673/TCP   14m    app.kubernetes.io/name=production-ready
NAME                                       READY   AGE   CONTAINERS   IMAGES
statefulset.apps/production-ready-server   4/4     24m   rabbitmq     rabbitmq:3.8.21-management
NAME                                            ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/production-ready   True               True               24m
[root@k8s-master production-ready]# kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-production-ready-server-0   Bound    pvc-71d4c7b8-554c-4f9e-856e-325e80ed88f2   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-1   Bound    pvc-6be2c9b3-a0b0-43ae-af1f-74ae92bce5ce   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-2   Bound    pvc-2243a65e-f38e-4164-a2f1-ec0a65be486d   12Gi       RWO            rook-ceph-block   27m
persistence-production-ready-server-3   Bound    pvc-26fca32f-d0d9-44af-8378-1a2aeefb1d94   12Gi       RWO            rook-ceph-block   3m21s

缩容不起作用。

7.Spring Boot测试

7.1.服务配置

msa-k8s-rabbitmq.yaml

apiVersion: v1
kind: Service
metadata:namespace: dev
  name: msa-k8s-rabbitmq
  labels:app: msa-k8s-rabbitmq
spec:type: ClusterIP
  ports:-port:8143targetPort:8143selector:app: msa-k8s-rabbitmq
---apiVersion: apps/v1
kind: Deployment
metadata:namespace: dev
  name: msa-k8s-rabbitmq
spec:replicas:3selector:matchLabels:app: msa-k8s-rabbitmq
  template:metadata:labels:app: msa-k8s-rabbitmq
    spec:imagePullSecrets:-name: harborsecret   #对应创建私有镜像密钥Secretcontainers:-name: msa-k8s-redis
          image: 172.51.216.85:8888/springcloud/msa-k8s-rabbitmq:1.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresentports:-containerPort:8143env:-name: ACTIVE
              value:"-Dspring.profiles.active=dev"

application-k8s.yml

# Redis集群配置server:port:8143spring:application:name: msa-k8s-rabbitmq
  #配置rabbitMq 服务器rabbitmq:host: production-ready.default.svc.cluster.local
    port:5672username: default_user_IdbGSVeW1plrpskejhI
    password: mYaPNH_Hlb-g1rNYMiPWGdKex-3l8po5
    #虚拟host 可以不设置,使用server默认host#virtual-host: JCcccHost# 说明# 集群内部访问：
production-ready
#完整配置host: production-ready.default.svc.cluster.local
# 或host: rabbitmq-cluster-svc.default.svc.cluster.local
[root@k8s-master rabbitmq]# kubectl get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
production-ready         ClusterIP   10.102.238.59   <none>        5672/TCP,15672/TCP,15692/TCP     89m
production-ready-nodes   ClusterIP   None            <none>        4369/TCP,25672/TCP               89m
rabbitmq-cluster-svc     NodePort    10.106.36.141   <none>        15672:30672/TCP,5672:30673/TCP   145m
# 集群外部访问，需要每个实例配置一个service,暴露端口# 参考erver:port:8143spring:application:name: msa-k8s-rabbitmq
  rabbitmq:# 集群地址，用逗号分隔addresses: 192.168.11.71:5672,192.168.11.72:5672,192.168.11.71:5673connection-timeout:15000password: guest
    # 使用启用消息确认模式publisher-confirms:trueusername: guest
    virtual-host: /
 
 
 # 配置每个实例的service  addresses: 192.168.11.71:5672,192.168.11.72:5672,192.168.11.71:5673

7.2.创建服务

docker build -t172.51.216.85:8888/springcloud/msa-k8s-rabbitmq:1.1.0 .docker push 172.51.216.85:8888/springcloud/msa-k8s-rabbitmq:1.1.0
[root@k8s-master operator]# kubectl apply -f msa-k8s-rabbitmq.yaml 
service/msa-k8s-redis created
deployment.apps/msa-k8s-redis created
root@k8s-master rabbitmq]# kubectl get all -n dev | grep msa-k8s-rabbitmq
pod/msa-k8s-rabbitmq-84976bbd47-8mh6p      1/1     Running            0          8m25s
pod/msa-k8s-rabbitmq-84976bbd47-9swjn      1/1     Running            0          8m25s
pod/msa-k8s-rabbitmq-84976bbd47-nsqbt      1/1     Running            0          8m25s
service/msa-k8s-rabbitmq         ClusterIP   10.107.32.2      <none>8143/TCP          8m25s
deployment.apps/msa-k8s-rabbitmq      3/3     33           8m25s
replicaset.apps/msa-k8s-rabbitmq-84976bbd47      333       8m25s

# 测试curl10.107.32.2:8143/send
[root@k8s-master rabbitmq]# curl 10.107.32.2:8143/send
发送消息成功！
[root@k8s-master rabbitmq]# curl 10.107.32.2:8143/send
发送消息成功！
[root@k8s-master rabbitmq]# kubectl logs -f msa-k8s-rabbitmq-84976bbd47-8mh6p
Error from server (NotFound): pods "msa-k8s-rabbitmq-84976bbd47-8mh6p" not found
[root@k8s-master rabbitmq]# kubectl logs -f msa-k8s-rabbitmq-84976bbd47-8mh6p -n dev
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=512M; support was removed in8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in8.0.   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ |'_| | '_ \/ _` |\\\\\\/  ___)||_)|||||||(_||))))'  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v2.3.10.RELEASE)
2021-12-01 14:15:22.184  INFO 1 --- [           main] com.k8s.msa.MsaK8sRabbitmqApplication    : Starting MsaK8sRabbitmqApplication v1.0.0 on msa-k8s-rabbitmq-84976bbd47-8mh6p with PID 1 (/app.jar started by root in /)
2021-12-01 14:15:22.189  INFO 1 --- [           main] com.k8s.msa.MsaK8sRabbitmqApplication    : The following profiles are active: dev
2021-12-01 14:15:23.880  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8143 (http)
2021-12-01 14:15:23.896  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-12-01 14:15:23.896  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.45]
2021-12-01 14:15:23.999  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-12-01 14:15:23.999  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1721 ms
2021-12-01 14:15:24.898  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2021-12-01 14:15:25.349  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8143 (http) with context path ''
2021-12-01 14:15:25.351  INFO 1 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: [production-ready.default.svc.cluster.local:5672]2021-12-01 14:15:25.447  INFO 1 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Created new connection: rabbitConnectionFactory#185a6e9:0/SimpleConnection@6cd28fa7 [delegate=amqp://default_user_IdbGSVeW1plrpskejhI@10.102.238.59:5672/, localPort= 33022]2021-12-01 14:15:25.530  INFO 1 --- [           main] com.k8s.msa.MsaK8sRabbitmqApplication    : Started MsaK8sRabbitmqApplication in4.14 seconds (JVM running for4.824)2021-12-01 14:16:30.850  INFO 1 --- [nio-8143-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]: Initializing Spring DispatcherServlet 'dispatcherServlet'2021-12-01 14:16:30.850  INFO 1 --- [nio-8143-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'2021-12-01 14:16:30.856  INFO 1 --- [nio-8143-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in6 ms
发送消息成功！：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:30 CST 2021
发送消息成功！：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:32 CST 2021
消费者消费消息了：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:32 CST 2021
发送消息成功！：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:35 CST 2021
消费者消费消息了：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:35 CST 2021
发送消息成功！：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:38 CST 2021
消费者消费消息了：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:40 CST 2021
发送消息成功！：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:41 CST 2021
消费者消费消息了：send，this is first messge！ $$$ -----V2Wed Dec 01 14:16:44 CST 2021

7.3.总结

1.开发测试：

每个RabbitMQ实例单独配置service，以NodePort方式暴露端口

2.集群内服访问

# Spring Boot# 集群内部访问：
production-ready
#完整配置
host: production-ready.default.svc.cluster.local
# 或
host: rabbitmq-cluster-svc.default.svc.cluster.local

8.删除集群

# 删除Rabbitmq[root@k8s-master production-ready]# pwd
/k8s/middleware/rabbitmq/cluster-operator/docs/examples/production-ready
[root@k8s-master production-ready]# ll
total 16
-rw-r--r-- 1 root root  199 Nov 3011:52 pod-disruption-budget.yaml
-rw-r--r-- 1 root root  500 Dec  112:14 rabbitmq.yaml
-rw-r--r-- 1 root root 3134 Nov 3011:52 README.md
-rw-r--r-- 1 root root  409 Nov 3011:52 ssd-gke.yaml
# 删除
kubectl delete -f pod-disruption-budget.yaml
kubectl delete -f rabbitmq.yaml
# 删除operator
kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

# 查看[root@k8s-master production-ready]# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7cbf865f89-s95gq   1/1     Running   0          4h31m
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     11           4h31m
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7cbf865f89   111       4h31m
[root@k8s-master production-ready]# kubectl get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/production-ready-server-0   1/1     Running   0          102m
pod/production-ready-server-1   1/1     Running   0          102m
pod/production-ready-server-2   1/1     Running   0          102m
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
service/kubernetes               ClusterIP   10.96.0.1       <none>443/TCP                          106d
service/production-ready         ClusterIP   10.102.238.59   <none>5672/TCP,15672/TCP,15692/TCP     102m
service/production-ready-nodes   ClusterIP   None            <none>4369/TCP,25672/TCP               102m
service/rabbitmq-cluster-svc     NodePort    10.106.36.141   <none>15672:30672/TCP,5672:30673/TCP   158m
NAME                                       READY   AGE
statefulset.apps/production-ready-server   3/3     102m
NAME                                            ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/production-ready   True               True               102m
[root@k8s-master production-ready]# kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistence-production-ready-server-0   Bound    pvc-0c3c4767-669c-497e-a6a7-c6ee4b6ff283   12Gi       RWO            rook-ceph-block   102m
persistence-production-ready-server-1   Bound    pvc-8ecc6b70-26ff-483d-a456-2325dd4376b3   12Gi       RWO            rook-ceph-block   102m
persistence-production-ready-server-2   Bound    pvc-6a9f3349-0396-46c1-a7a9-7fd8ad9c218b   12Gi       RWO            rook-ceph-block   102m
# 删除# 删除svc[root@k8s-master rabbitmq]# kubectl delete -f rabbitmq-cluster-svc.yaml service"rabbitmq-cluster-svc" deleted
# 删除rabbitmq[root@k8s-master production-ready]# kubectl delete -f pod-disruption-budget.yaml 
poddisruptionbudget.policy "production-ready-rabbitmq" deleted
[root@k8s-master production-ready]# kubectl delete -f rabbitmq.yaml 
rabbitmqcluster.rabbitmq.com "production-ready" deleted
[root@k8s-master rabbitmq]# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>443/TCP   105d
# 删除operator[root@k8s-master rabbitmq]# kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
namespace "rabbitmq-system" deleted
customresourcedefinition.apiextensions.k8s.io "rabbitmqclusters.rabbitmq.com" deleted
serviceaccount "rabbitmq-cluster-operator" deleted
role.rbac.authorization.k8s.io "rabbitmq-cluster-leader-election-role" deleted
clusterrole.rbac.authorization.k8s.io "rabbitmq-cluster-operator-role" deleted
rolebinding.rbac.authorization.k8s.io "rabbitmq-cluster-leader-election-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "rabbitmq-cluster-operator-rolebinding" deleted
deployment.apps "rabbitmq-cluster-operator" deleted
[root@k8s-master rabbitmq]# kubectl get all -n rabbitmq-system
No resources found in rabbitmq-system namespace.
[root@k8s-master rabbitmq]# kubectl get pvc
No resources found in default namespace.

• Kubernetes

# Kubernetes
https://iothub.org.cn/docs/kubernetes/
https://iothub.org.cn/docs/kubernetes/middleware/rabbitmq/