Flink CDC实践

Flink CDC实践

使用 Flink CDC 3.0 ELT 从 MySQL 流传输到 Doris

配置环境

参考https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/mysql-doris-pipeline-tutorial.html

前提：安装JDK环境！！！
参考：Centos7下安装java8

1. 下载flink安装包 从官网https://flink.apache.org/downloads/下载flink安装包，网速较慢，可以尝试在国内镜像网址https://mirrors.cloud.tencent.com/apache/flink下载，上传到服务器
2. 解压flink安装包mkdir /usr/local/flinktar-zxvf /opt/flink-1.16.0-bin-scala_2.12.tgz -C /usr/local/flink
3. 配置flink环境变量sudovi /etc/profile.d/myprofile.shexportFLINK_HOME=/usr/local/flink/flink-1.16.0``````source /etc/profile
4. 关闭防火墙，配置systemctl status firewalldsystemctl stop firewalldsystemctl disable firewalld //禁止开机启动``````cd /usr/local/flink/flink-1.16.0/confvi flink-conf.yaml配置Rest & web frontend项：rest.bind-address: 0.0.0.0
5. 启动flink，查看启动是否成功cd /usr/local/flink/flink-1.16.0/bin./start-cluster.shjps //查看是否启动成功访问http://虚拟机ip:8081，查看flink webUI是否正常访问ifconfig虚拟机ip查看： 备注：如果输入ifconfig没有找到命令执行：yum search ifconfigyum install net-tools.x86_64

使用 FlinkCDC cli 提交作业

1. 解压flink-cdc安装包tar-zxvf /opt/flink-cdc-3.0.0-bin.tar.gz -C /usr/local/flink
2. 将mysql和doris的驱动包移动到lib目录下mv /opt/flink-cdc-pipeline-connector-mysql-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/libmv /opt/flink-cdc-pipeline-connector-doris-3.0.0.jar /usr/local/flink/flink-cdc-3.0.0/lib
3. 配置yaml文件cd /usr/local/flink/flink-cdc-3.0.0/mkdir pipelinevi ./pipeline/mysqlcdc_to_doris.yaml``````################################################################################# Description: Sync MySQL all tables to Doris################################################################################ source: type: mysql hostname: localhost port: 3306 username: xxx password: xxx tables: app_db.\.* server-id: 5400-5404 server-time-zone: UTC sink: type: doris fenodes: 127.0.0.1:8030 username: root password: "" table.create.properties.light_schema_change: true table.create.properties.replication_num: 1 pipeline: name: Sync MySQL Database to Doris parallelism: 2
4. 启动cd /usr/local/flink/flink-cdc-3.0.0/bash ./bin/flink-cdc.sh ./pipeline/mysqlcdc_to_doris.yaml

出现的问题

1. flinksql-Could not acquire the minimum required resources 无法获取所需的最低资源，其实是可用任务槽太少，任务过多时，任务槽就不够用了 解决方法：修改配置文件 flink-conf.yaml，重新设置任务槽的数量cd /usr/local/flink/flink-1.16.0/confvi flink-conf.yamltaskmanager.numberOfTaskSlots: 30