0


Flink CDC实践

Flink CDC实践

使用 Flink CDC 3.0 ELT 从 MySQL 流传输到 Doris

配置环境

参考https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/mysql-doris-pipeline-tutorial.html

前提:安装JDK环境!!!
参考:Centos7下安装java8

  1. 下载flink安装包 从官网https://flink.apache.org/downloads/下载flink安装包,网速较慢,可以尝试在国内镜像网址https://mirrors.cloud.tencent.com/apache/flink下载,上传到服务器![在这里插入图片描述](https://img-blog.csdnimg.cn/direct/81e637a262fd4c8692e02bfe5806dda9.png)
  2. 解压flink安装包mkdir /usr/local/flinktar-zxvf /opt/flink-1.16.0-bin-scala_2.12.tgz -C /usr/local/flink
  3. 配置flink环境变量sudovi /etc/profile.d/myprofile.shexportFLINK_HOME=/usr/local/flink/flink-1.16.0``````source /etc/profile
  4. 关闭防火墙,配置systemctl status firewalldsystemctl stop firewalldsystemctl disable firewalld //禁止开机启动``````cd /usr/local/flink/flink-1.16.0/confvi flink-conf.yaml配置Rest & web frontend项:rest.bind-address: 0.0.0.0
  5. 启动flink,查看启动是否成功cd /usr/local/flink/flink-1.16.0/bin./start-cluster.shjps //查看是否启动成功访问http://虚拟机ip:8081,查看flink webUI是否正常访问ifconfig虚拟机ip查看:在这里插入图片描述 备注:如果输入ifconfig没有找到命令执行:yum search ifconfigyum install net-tools.x86_64在这里插入图片描述

使用 FlinkCDC cli 提交作业

  1. 解压flink-cdc安装包tar-zxvf /opt/flink-cdc-3.0.0-bin.tar.gz -C /usr/local/flink
  2. 将mysql和doris的驱动包移动到lib目录下mv /opt/flink-cdc-pipeline-connector-mysql-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/libmv /opt/flink-cdc-pipeline-connector-doris-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/lib
  3. 配置yaml文件cd /usr/local/flink/flink-cdc-3.0.0/mkdir pipelinevi ./pipeline/mysqlcdc_to_doris.yaml``````################################################################################# Description: Sync MySQL all tables to Doris################################################################################ source: type: mysql hostname: localhost port: 3306 username: xxx password: xxx tables: app_db.\.* server-id: 5400-5404 server-time-zone: UTC sink: type: doris fenodes: 127.0.0.1:8030 username: root password: "" table.create.properties.light_schema_change: true table.create.properties.replication_num: 1 pipeline: name: Sync MySQL Database to Doris parallelism: 2
  4. 启动cd /usr/local/flink/flink-cdc-3.0.0/bash ./bin/flink-cdc.sh ./pipeline/mysqlcdc_to_doris.yaml

出现的问题

  1. flinksql-Could not acquire the minimum required resources 无法获取所需的最低资源,其实是可用任务槽太少,任务过多时,任务槽就不够用了 解决方法:修改配置文件 flink-conf.yaml,重新设置任务槽的数量cd /usr/local/flink/flink-1.16.0/confvi flink-conf.yamltaskmanager.numberOfTaskSlots: 30
标签: flink 大数据

本文转载自: https://blog.csdn.net/wwplcy/article/details/135743022
版权归原作者 云朵儿的bolg 所有, 如有侵权,请联系我们删除。

“Flink CDC实践”的评论:

还没有评论