skywalking使用kafka传输数据

集群配置说明

集群模式

• 集群模式- 基于nacos
• 动态配置发布- 基于nacos
• 存储- 基于elasticsearch7
• 数据传输方式: kafka队列

nacos配置参照nacos部署

es配置参照ELK部署

服务分布

服务模式IP用途skywalking-OAP集群3.1.20.23, 3.1.20.24后端skywalking-UI单点3.1.20.32ui页面kafka集群3.1.20.28, 3.1.20.29, 3.1.20.30队列es集群3.1.20.25, 3.1.20.26, 3.1.20.27存储Nginx单点3.1.20.85做sw-oap 11800端口和12800端口的负载

docker-compose方式部署

• 根据配置文件, 直接使用compose方式注册变量

SW-OAP

• es7存储
• nacos注册中心
• nacos动态配置

# 创建目录
mkdir -pv /data/docker-compose/skywalking

# 创建docker-compose编排文件
cat > /data/docker-compose/skywalking/docker-compose.yml << EOF
version:"3"services:oap:container_name: sw-oap
    image: 3.1.101.57:8005/public/skywalking-oap-server:8.6.0-es7
    restart: always
    network_mode: host
    volumes:- /etc/localtime:/etc/localtime
      - /data/skywalking/data:/root
    environment:## 集群配置-nacosSW_CLUSTER: nacos
      SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"SW_CLUSTER_NACOS_HOST_PORT: 3.1.20.85:8848SW_CLUSTER_NACOS_NAMESPACE:"skywalking"SW_CLUSTER_NACOS_USERNAME:"nacos"SW_CLUSTER_NACOS_PASSWORD:"Nac0s#123456!"## 存储配置-es7SW_STORAGE: elasticsearch7
      SW_STORAGE_ES_CLUSTER_NODES: 3.1.20.25:9200,3.1.20.26:9200,3.1.20.27:9200SW_ES_USER: elastic
      SW_ES_PASSWORD: G1T@es2022#ccms## core配置SW_CORE_REST_HOST: 3.1.20.23
      SW_CORE_REST_PORT:12800SW_CORE_GRPC_HOST: 3.1.20.23
      SW_CORE_GRPC_PORT:11800## 动态配置发布-nacosSW_CONFIGURATION: nacos
      SW_CONFIG_NACOS_SERVER_ADDR: 3.1.20.85
      SW_CONFIG_NACOS_SERVER_PORT:8848SW_CONFIG_NACOS_SERVER_GROUP: skywalking
      SW_CONFIG_NACOS_SERVER_NAMESPACE: skywalking
      SW_CONFIG_NACOS_USERNAME:"nacos"SW_CONFIG_NACOS_PASSWORD:"Nac0s#123456!"## 使用kafka传输数据SW_KAFKA_FETCHER:'default'SW_KAFKA_FETCHER_SERVERS:'3.1.20.28:9092,3.1.20.29:9092,3.1.20.30:9092'SW_NAMESPACE:'topic-dics-long'SW_KAFKA_FETCHER_PARTITIONS:"3"SW_KAFKA_FETCHER_PARTITIONS_FACTOR:"2"SW_KAFKA_FETCHER_ENABLE_METER_SYSTEM:"true"TZ: Asia/Shanghai
      JAVA_OPTS:"-Xms12G -Xmx12G"deploy:resources:limits:memory: 24G
EOF

所有变量可通过配置文件查询, 容器内配置文件位置: /skywalking/config/application.yml

SW-UI

# 创建目录
mkdir -pv /data/docker-compose/skywalking

# 创建docker-compose编排文件
cat > /data/docker-compose/skywalking/docker-compose.yml << EOF
version:"3"services:ui:container_name: sw-ui
    image: 3.1.101.57:8005/public/skywalking-ui:8.6.0
    ports:- 12808:8080restart: always
    volumes:- /etc/localtime:/etc/localtime
    environment:TZ: Asia/Shanghai
      SW_OAP_ADDRESS: 3.1.20.23:12800,3.1.20.24:12800deploy:resources:limits:memory: 2G
EOF

Agent配置

配置agent插件

使用kafka传输数据, 需要启用kafka上报插件

cd skywalking/agent/optional-reporter-plugins;mv kafka-reporter-plugin-8.6.0.jar ../plugins/

agent配置文件

agent.service_name=${SW_AGENT_NAME:credit-gateway}collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:3.1.20.23:11800,3.1.20.24:11800}logging.file_name=${SW_LOGGING_FILE_NAME:skywalking-api.log}logging.level=${SW_LOGGING_LEVEL:INFO}plugin.mount=${SW_MOUNT_FOLDERS:plugins,activations}plugin.jdbc.trace_sql_parameters=${SW_JDBC_TRACE_SQL_PARAMETERS:true}plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:3.1.20.85:11800}plugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:11800}plugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:10485760}plugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}plugin.kafka.bootstrap_servers=${SW_KAFKA_BOOTSTRAP_SERVERS:3.1.20.28:9092,3.1.20.29:9092,3.1.20.30:9092}plugin.kafka.namespace=${SW_KAFKA_NAMESPACE:"topic-dics-long"}