1、组件环境
名称版本描述下载地址hadoop3.4.0官方下载binhive3.1.3下载源码编译mysql8.0.31datax0.0.1-SNAPSHOThttp://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gzdatax-webdatax-web-2.1.2下载源码进行编译github.comcentoscentos7x86版本java1.8
2、安装datax
2.1、下载datax并解压
http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
tar -zxvf datax.tar.gz
解压到/cluster目录下
执行测试命令
./bin/datax.py job/job.json
出现报错
File "/cluster/datax/bin/./datax.py", line 114
print readerRef
^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?说明：系统中安装python3和python2，默认是python3，由于datax bin中的datax默认支持python2，
所以需要指定python版本为python2，后面会将这个三个文件进行备份替换，datax-web中doc下提供了默认的支持。
- Python (2.x) (支持Python3需要修改替换datax/bin下面的三个python文件，替换文件在doc/datax-web/datax-python3下) 必选，主要用于调度执行底层DataX的启动脚本，默认的方式是以Java子进程方式执行DataX，用户可以选择以Python方式来做自定义的改造
- 参考地址：datax-web/doc/datax-web/datax-web-deploy.md at master · WeiYe-Jing/datax-web (github.com)
python2 bin/datax.py job/job.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.2024-10-13 21:04:41.543 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-10-13 21:04:41.553 [main] INFO Engine - the machine info =>osInfo: Oracle Corporation 1.8 25.40-b25 jvmInfo: Linux amd64 5.8.13-1.el7.elrepo.x86_64 cpu num: 8 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB2024-10-13 21:04:41.575 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"streamreader",
"parameter":{
"column":[
{
"type":"string",
"value":"DataX"
},
{
"type":"long",
"value":19890604
},
{
"type":"date",
"value":"1989-06-04 00:00:00"
},
{
"type":"bool",
"value":true
},
{
"type":"bytes",
"value":"test"
}
],
"sliceRecordCount":100000
}
},
"writer":{
"name":"streamwriter",
"parameter":{
"encoding":"UTF-8",
"print":false
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0.02,
"record":0
},
"speed":{
"byte":10485760
}
}
}2024-10-13 21:04:41.599 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2024-10-13 21:04:41.601 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-10-13 21:04:41.601 [main] INFO JobContainer - DataX jobContainer starts job.
2024-10-13 21:04:41.604 [main] INFO JobContainer - Set jobId = 0
2024-10-13 21:04:41.623 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2024-10-13 21:04:41.624 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do prepare work .
2024-10-13 21:04:41.624 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2024-10-13 21:04:41.624 [job-0] INFO JobContainer - jobContainer starts to do split ...
2024-10-13 21:04:41.625 [job-0] INFO JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2024-10-13 21:04:41.626 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2024-10-13 21:04:41.627 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2024-10-13 21:04:41.649 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2024-10-13 21:04:41.654 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-10-13 21:04:41.657 [job-0] INFO JobContainer - Running by standalone Mode.
2024-10-13 21:04:41.666 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2024-10-13 21:04:41.671 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2024-10-13 21:04:41.672 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-10-13 21:04:41.685 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2024-10-13 21:04:41.986 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[302]ms
2024-10-13 21:04:41.987 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2024-10-13 21:04:51.677 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.022s | All Task WaitReaderTime 0.040s | Percentage 100.00%
2024-10-13 21:04:51.677 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2024-10-13 21:04:51.678 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do post work.
2024-10-13 21:04:51.678 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do post work.
2024-10-13 21:04:51.678 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2024-10-13 21:04:51.680 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /cluster/datax/hook
2024-10-13 21:04:51.682 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%[total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s2024-10-13 21:04:51.682 [job-0] INFO JobContainer - PerfTrace not enable!
2024-10-13 21:04:51.683 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.022s | All Task WaitReaderTime 0.040s | Percentage 100.00%
2024-10-13 21:04:51.684 [job-0] INFO JobContainer -
任务启动时刻 : 2024-10-13 21:04:41
任务结束时刻 : 2024-10-13 21:04:51
任务总计耗时 : 10s
任务平均流量 : 253.91KB/s
记录写入速度 : 10000rec/s
读出记录总数 : 100000
读写失败总数 : 0
./bin/datax.py -r hdfsreader -w mysqlwriter
3、安装datax-web
3.0、下载datax-web的源码，进行编译
git@github.com:WeiYe-Jing/datax-web.git
mvn -U clean package assembly:assembly -Dmaven.test.skip=true
- 打包成功后的DataX包位于 {DataX_source_code_home}/target/datax/datax/ ，结构如下：
$ cd {DataX_source_code_home}$ ls ./target/datax/datax/bin conf job lib log log_perf plugin script tmp
3.1、在MySQL中创建datax-web元数据
mysql -u root -p
password:******
CREATE DATABASE dataxweb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
use dataxweb;
3.2、安装data-web
3.2.1执行install.sh命令解压部署
在/cluster/datax-web-2.1.2/bin目录下
执行./install.sh
根据提示执行
执行完成后，会解压文件，并初始化数据库。
3.2.1、手动修改 datax-admin配置文件
/cluster/datax-web-2.1.2/modules/datax-admin/bin/env.properties
内容如下
environment variables
JAVA_HOME=/java/jdk
WEB_LOG_PATH=/cluster/datax-web-2.1.2/modules/datax-admin/logs
WEB_CONF_PATH=/cluster/datax-web-2.1.2/modules/datax-admin/confDATA_PATH=/cluster/datax-web-2.1.2/modules/datax-admin/data
SERVER_PORT=6895PID_FILE_PATH=/cluster/datax-web-2.1.2/modules/datax-admin/dataxadmin.pid
mail account
MAIL_USERNAME="1024122298@qq.com"
MAIL_PASSWORD="*********************"#debug
REMOTE_DEBUG_SWITCH=true
REMOTE_DEBUG_PORT=7223
3.2.2、手动修改 datax-executor配置文件
/cluster/datax-web-2.1.2/modules/datax-executor/bin/env.properties
内容如下：主要是PYTHON_PATH=/cluster/datax/bin/datax.py
environment variables
JAVA_HOME=/java/jdk
SERVICE_LOG_PATH=/cluster/datax-web-2.1.2/modules/datax-executor/logs
SERVICE_CONF_PATH=/cluster/datax-web-2.1.2/modules/datax-executor/conf
DATA_PATH=/cluster/datax-web-2.1.2/modules/datax-executor/datadatax json文件存放位置
JSON_PATH=/cluster/datax-web-2.1.2/modules/datax-executor/json
executor_port
EXECUTOR_PORT=9999
保持和datax-admin端口一致
DATAX_ADMIN_PORT=6895
PYTHON脚本执行位置
PYTHON_PATH=/cluster/datax/bin/datax.py
dataxweb 服务端口
SERVER_PORT=9504
PID_FILE_PATH=/cluster/datax-web-2.1.2/modules/datax-executor/service.pid
#debug 远程调试端口
REMOTE_DEBUG_SWITCH=true
REMOTE_DEBUG_PORT=7224
3.2.3、替换datax下的python执行文件
- Python (2.x) (支持Python3需要修改替换datax/bin下面的三个python文件，替换文件在doc/datax-web/datax-python3下) 必选，主要用于调度执行底层DataX的启动脚本，默认的方式是以Java子进程方式执行DataX，用户可以选择以Python方式来做自定义的改造
- 这个单个文件可从datax-web源码目录中获取
3.2.4、替换MySQLjar文件
datax下的mysql reader和writer的jar版本过低，使用的mysql8数据库，需要替换下jar文件
路径是：
/cluster/datax/plugin/writer/mysqlwriter/libs/mysql-connector-j-8.3.0.jar
和
/cluster/datax/plugin/reader/mysqlreader/libs/mysql-connector-j-8.3.0.jar
4、创建MySQL和Hive数据
4.1、创建MySQL数据库
=================================================================================================================================
2024年10月13日 星期日 第41周 00:07:20
mysql建表
-- m31094.mm definition
=================================================================================================================================
CREATE TABLEmm(
uuidvarchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
namevarchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
timevarchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
agevarchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
sexvarchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
jobvarchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
addresstext COLLATE utf8mb4_unicode_ci,
PRIMARY KEY (uuid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;不设置主键uuid
4.2、创建Hive数据库
hive建表
create database m31094;
drop table m31094.mm;
CREATE TABLE m31094.mm (
uuidSTRING COMMENT '主键',
nameSTRING COMMENT '姓名',
timeSTRING COMMENT '时间',
ageSTRING COMMENT '年龄',
sexSTRING COMMENT '性别',
jobSTRING COMMENT '工作',
addressSTRING COMMENT '地址'
) COMMENT '美女表'
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
建表信息：
hive查询表描述信息
DESCRIBE FORMATTED m31094.mm;
5、配置datax和datax-web
5.1、启动datax-web服务
启动命令
/cluster/datax-web-2.1.2/modules/datax-admin/bin/datax-admin.sh restart
tail -f /cluster/datax-web-2.1.2/modules/datax-admin/bin/console.out
/cluster/datax-web-2.1.2/modules/datax-executor/bin/datax-executor.sh restart
tail -f /cluster/datax-web-2.1.2/modules/datax-executor/bin/console.out
查看进程的命令jps -l
kill进程的命令
sudo kill -9 $(ps -ef|grep datax|gawk '$0 !~/grep/ {print $2}' |tr -s '\n' ' ')
datax-executor需要启动成功，可在页面查看自动注册的信息。
5.2、页面访问及配置
6895是自定义端口，根据实际设置
登录用户名、密码:admin/123456
5.2.1创建项目
5.2.2、添加mysql和hive的数据源
MySQL
HIVE
5.2.3创建DataX任务模板
5.2.4、任务构建
步骤一、配置输入
步骤二、配置输出
步骤三、 字段映射
步骤四、构建、选择任务模板，复制JSON、下一步操作
生成JSON模板内容
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "/cluster/hive/warehouse/m31094.db/mm",
"defaultFS": "hdfs://10.7.215.33:8020",
"fileType": "text",
"fieldDelimiter": ",",
"skipHeader": false,
"column": [
{
"index": "0",
"type": "string"
},
{
"index": "1",
"type": "string"
},
{
"index": "2",
"type": "string"
},
{
"index": "3",
"type": "string"
},
{
"index": "4",
"type": "string"
},
{
"index": "5",
"type": "string"
},
{
"index": "6",
"type": "string"
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "yRjwDFuoPKlqya9h9H2Amg==",
"password": "XCYVpFosvZBBWobFzmLWvA==",
"column": [
"uuid",
"name",
"time",
"age",
"sex",
"job",
"address"
],
"connection": [
{
"table": [
"mm"
],
"jdbcUrl": "jdbc:mysql://10.7.215.33:3306/m31094"
}
]
}
}
}
]
}
}
5.2.5、查询创建的任务
5.2.6、在Hive中执行数据插入
insert into m31094.mm values('9','hive数据使用datax同步到MySQL',from_unixtime(unix_timestamp()),'1000000000090101','北京','新疆','加油');
控制台输出内容：
6、查看效果
日志执行情况
2024-10-13 21:30:00 [JobThread.run-130] <br>----------- datax-web job execute start -----------<br>----------- Param: 2024-10-13 21:30:00 [BuildCommand.buildDataXParam-100] ------------------Command parameters: 2024-10-13 21:30:00 [ExecutorJobHandler.execute-57] ------------------DataX process id: 95006 2024-10-13 21:30:00 [ProcessCallbackThread.callbackLog-186] <br>----------- datax-web job callback finish. 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.588 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.597 [main] INFO Engine - the machine info => 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] osInfo: Oracle Corporation 1.8 25.40-b25 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] jvmInfo: Linux amd64 5.8.13-1.el7.elrepo.x86_64 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] cpu num: 8 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] totalPhysicalMemory: -0.00G 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] freePhysicalMemory: -0.00G 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] maxFileDescriptorCount: -1 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] currentOpenFileDescriptorCount: -1 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] GC Names [PS MarkSweep, PS Scavenge] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] MEMORY_NAME | allocation_size | init_size 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] PS Eden Space | 256.00MB | 256.00MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] Code Cache | 240.00MB | 2.44MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] Compressed Class Space | 1,024.00MB | 0.00MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] PS Survivor Space | 42.50MB | 42.50MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] PS Old Gen | 683.00MB | 683.00MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] Metaspace | -0.00MB | 0.00MB 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.622 [main] INFO Engine - 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "content":[ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "reader":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "name":"hdfsreader", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "column":[ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"0", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"1", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"2", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"3", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"4", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"5", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "index":"6", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] ], 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "defaultFS":"hdfs://10.7.215.33:8020", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "fieldDelimiter":",", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "fileType":"text", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "path":"hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "skipHeader":false 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "writer":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "name":"mysqlwriter", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "column":[ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`uuid`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`name`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`time`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`age`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`sex`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`job`", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "`address`" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] ], 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "connection":[ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] { 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "jdbcUrl":"jdbc:mysql://10.7.215.33:3306/m31094", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "table":[ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "mm" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] ] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] ], 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "password":"******", 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "username":"root" 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] ], 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "setting":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "errorLimit":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "percentage":0.02, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "record":0 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] }, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "speed":{ 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "byte":1048576, 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] "channel":3 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] } 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.645 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.647 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.648 [main] INFO JobContainer - DataX jobContainer starts job. 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.650 [main] INFO JobContainer - Set jobId = 0 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.667 [job-0] INFO HdfsReader$Job - init() begin... 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.993 [job-0] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":[]} 2024-10-13 21:30:00 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:00.994 [job-0] INFO HdfsReader$Job - init() ok and end... 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.580 [job-0] INFO OriginalConfPretreatmentUtil - table:[mm] all columns:[ 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] uuid,name,time,age,sex,job,address 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] ]. 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.613 [job-0] INFO OriginalConfPretreatmentUtil - Write data [ 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] INSERT INTO %s (`uuid`,`name`,`time`,`age`,`sex`,`job`,`address`) VALUES(?,?,?,?,?,?,?) 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] ], which jdbcUrl like:[jdbc:mysql://10.7.215.33:3306/m31094?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true] 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.614 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.614 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do prepare work . 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.614 [job-0] INFO HdfsReader$Job - prepare(), start to getAllFiles... 2024-10-13 21:30:01 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:01.614 [job-0] INFO HdfsReader$Job - get HDFS all files in path = [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm] 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.699 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.709 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_1]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.718 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_2]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.728 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_3]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.737 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_4]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.759 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_5]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.771 [job-0] INFO HdfsReader$Job - [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_6]是[text]类型的文件, 将该文件加入source files列表 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.771 [job-0] INFO HdfsReader$Job - 您即将读取的文件数为: [7], 列表为: [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_5,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_4,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_3,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_2,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_1,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0,hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_6] 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.772 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work . 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.773 [job-0] INFO JobContainer - jobContainer starts to do split ... 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.774 [job-0] INFO JobContainer - Job set Max-Byte-Speed to 1048576 bytes. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.775 [job-0] INFO HdfsReader$Job - split() begin... 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.777 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] splits to [7] tasks. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.779 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [7] tasks. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.797 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.810 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.813 [job-0] INFO JobContainer - Running by standalone Mode. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.825 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [7] tasks. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.830 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.831 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.845 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.883 [0-0-2-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.885 [0-0-2-reader] INFO Reader$Task - read start 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.886 [0-0-2-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_3] 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.918 [0-0-2-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:02 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:02.925 [0-0-2-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.247 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[403]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.250 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.286 [0-0-5-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.287 [0-0-5-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.287 [0-0-5-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.290 [0-0-5-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.292 [0-0-5-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.351 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[101]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.354 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.379 [0-0-4-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.380 [0-0-4-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.380 [0-0-4-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_1] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.384 [0-0-4-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.386 [0-0-4-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.454 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[101]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.457 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.486 [0-0-0-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.487 [0-0-0-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.487 [0-0-0-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_5] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.489 [0-0-0-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.491 [0-0-0-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.558 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[101]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.561 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.587 [0-0-1-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.588 [0-0-1-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.588 [0-0-1-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_4] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.592 [0-0-1-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.594 [0-0-1-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.662 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[101]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.664 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.691 [0-0-3-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.691 [0-0-3-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.691 [0-0-3-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_2] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.694 [0-0-3-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.696 [0-0-3-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.765 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[101]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.768 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.791 [0-0-6-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.791 [0-0-6-reader] INFO Reader$Task - read start 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.791 [0-0-6-reader] INFO Reader$Task - reading file : [hdfs://10.7.215.33:8020/cluster/hive/warehouse/m31094.db/mm/000000_0_copy_6] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.795 [0-0-6-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.798 [0-0-6-reader] INFO Reader$Task - end read source files... 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.868 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[100]ms 2024-10-13 21:30:03 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:03.869 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.838 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 282 bytes | Speed 28B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.028s | Percentage 100.00% 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.838 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.838 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.839 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do post work. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.839 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.840 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /cluster/datax/hook 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.841 [job-0] INFO JobContainer - 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] [total cpu info] => 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] averageCpu | maxDeltaCpu | minDeltaCpu 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] -1.00% | -1.00% | -1.00% 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] [total gc info] => 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] PS MarkSweep | 1 | 1 | 1 | 0.040s | 0.040s | 0.040s 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] PS Scavenge | 1 | 1 | 1 | 0.022s | 0.022s | 0.022s 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.841 [job-0] INFO JobContainer - PerfTrace not enable! 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.842 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 282 bytes | Speed 28B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.028s | Percentage 100.00% 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12.843 [job-0] INFO JobContainer - 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 任务启动时刻 : 2024-10-13 21:30:00 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 任务结束时刻 : 2024-10-13 21:30:12 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 任务总计耗时 : 12s 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 任务平均流量 : 28B/s 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 记录写入速度 : 0rec/s 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 读出记录总数 : 7 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 读写失败总数 : 0 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 十月 13, 2024 9:30:01 下午 org.apache.hadoop.util.NativeCodeLoader <clinit> 2024-10-13 21:30:12 [AnalysisStatistics.analysisStatisticsLog-53] 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-10-13 21:30:12 [JobThread.run-165] <br>----------- datax-web job execute end(finish) -----------<br>----------- ReturnT:ReturnT [code=200, msg=LogStatistics{taskStartTime=2024-10-13 21:30:00, taskEndTime=2024-10-13 21:30:12, taskTotalTime=12s, taskAverageFlow=28B/s, taskRecordWritingSpeed=0rec/s, taskRecordReaderNum=7, taskRecordWriteFailNum=0}, content=null] 2024-10-13 21:30:12 [TriggerCallbackThread.callbackLog-186] <br>----------- datax-web job callback finish.
查询MySQL数据库查询数据同步
版权归原作者 青春不流名 所有， 如有侵权，请联系我们删除。