0 背景
本文基于Ambari集群搭建最新版本的海豚调度dolphinscheduler-3.1.3版本，后续会尝试整合到Ambari中。
1 安装准备
安装dolphinscheduler需要在环境中安装如下依赖
① JDK8
下载JDK (1.8+)，安装并配置 JAVA_HOME 环境变量，并将其下的 bin 目录追加到 PATH 环境变量中
② 数据库：PostgreSQL (8.2.15+) 或者 MySQL (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
③ 注册中心：ZooKeeper (3.4.6+)
2 安装集群
节点ip服务角色Ambari-1192.168.x.31ds-master ds-api ds-alertAmbari-2192.168.x.32ds-masterAmbari-3192.168.x.33ds-workerAmbari-4192.168.x.34ds-workerAmbari-5192.168.x.35ds-worker
2.1 配置dolphinscheduler用户
新建dolphinscheduler用户，并且一定要配置 sudo 免密.
# 创建用户需使用 root 登录useradd dolphinscheduler
# 添加密码echo"dolphinscheduler"|passwd--stdin dolphinscheduler
# 配置 sudo 免密sed-i'$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed-i's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
# 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限chown-R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
配置免密
su dolphinscheduler
ssh-keygen -t rsa -P''-f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod600 ~/.ssh/authorized_keys
集群之间完成免密
ssh-copy-id -i ~/.ssh/id_rsa.pub dolphinscheduler@主机
2.2 配置mysql
DolphinScheduler 元数据存储在关系型数据库中，故需创建相应的数据库和用户。注：这里根据Mysql安装版本的不同选择不同的执行方式。
mysql5.6/5.7
mysql -uroot -p
mysql>CREATEDATABASE dolphinscheduler DEFAULTCHARACTERSET utf8 DEFAULTCOLLATE utf8_general_ci;# 修改 {user} 和 {password} 为你希望的用户名和密码
mysql>GRANTALLPRIVILEGESON dolphinscheduler.*TO'{user}'@'%' IDENTIFIED BY'{password}';
mysql>GRANTALLPRIVILEGESON dolphinscheduler.*TO'{user}'@'localhost' IDENTIFIED BY'{password}';
mysql> flush privileges;
mysql8:
创建数据库并指定编码格式
mysql>CREATEDATABASE dolphinscheduler DEFAULTCHARACTERSET utf8 DEFAULTCOLLATE utf8_general_ci;
开启权限
# 修改 {user} 和 {password} 为你希望的用户名和密码
mysql>CREATEUSER'{user}'@'%' IDENTIFIED BY'{password}';
mysql>GRANTALLPRIVILEGESON dolphinscheduler.*TO'{user}'@'%';
mysql>CREATEUSER'{user}'@'localhost' IDENTIFIED BY'{password}';
mysql>GRANTALLPRIVILEGESON dolphinscheduler.*TO'{user}'@'localhost';
mysql> FLUSH PRIVILEGES;
执行过程中可能出现如下报错：
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
执行如下命令查看mysql密码要求规则
show variables like'validate_password%';
修改校验登记为LOW,重新设置即可
setglobal validate_password.policy=LOW;
2.3 启动zk
2.4 修改ds配置文件
2.4.1 mysql的JDBC驱动放入ds的lib目录下
包括 api-server/libs 和 alert-server/libs 和 master-server/libs 和 worker-server/libs
cp../mysql-connector-java-8.0.16.jar ./lib
cp../mysql-connector-java-8.0.16.jar ./api-server/libs/
cp../mysql-connector-java-8.0.16.jar ./alert-server/libs/
cp../mysql-connector-java-8.0.16.jar ./master-server/libs/
cp../mysql-connector-java-8.0.16.jar ./worker-server/libs/
2.4. 2 修改bin/env目录下的install_env.sh配置文件
我们仅需修改INSTALL MACHINE，DolphinScheduler ENV、Database、Registry Server
2.4.3 修改dolphinscheduler_env.sh 文件
bin/env/dolphinscheduler_env.sh 中配置JAVA_HOME、数据库、注册中心等相关配置
2.6 初始化数据库
bash tools/bin/upgrade-schema.sh
2.7 启动ds
sh install.sh
报错如下：
9:32:47.555 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
09:32:47.561 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
09:32:47.578 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
ZooKeeper -server host:port -client-configuration properties-file cmd args
addWatch [-m mode] path # optional mode is one of [PERSISTENT, PERSISTENT_RECURSIVE] - default is PERSISTENT_RECURSIVE
addauth scheme auth
close
config [-c] [-w] [-s]
connect host:port
create [-s] [-e] [-c] [-t ttl] path [data] [acl]
delete [-v version] path
deleteall path [-b batch size]
delquota [-n|-b|-N|-B] path
get [-s] [-w] path
getAcl [-s] path
getAllChildrenNumber path
getEphemerals path
history
listquota path
ls [-s] [-w] [-R] path
printwatches on|off
quit
reconfig [-s] [-v version] [[-file path] | [-members serverID=host:port1:port2;port3[,…]]] | [-add serverId=host:port1:port2;port3[,…]] [-remove serverId[,…]*]
redo cmdno
removewatches path [-c|-d|-a] [-l]
set [-s] [-v version] path data
setAcl [-s] [-v version] [-R] path acl
setquota -n|-b|-N|-B val path
stat [-w] path
sync path
version
whoami
Command not found: Command not found rmr
09:32:47.584 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 127
查看具体生成日志发现如下报错：
Caused by: org.apache.dolphinscheduler.registry.api.RegistryException: Failed to put registry key: /nodes/master
239 at org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperRegistry.put(ZookeeperRegistry.java:186)
240 at org.apache.dolphinscheduler.service.registry.RegistryClient.initNodes(RegistryClient.java:247)
241 at org.apache.dolphinscheduler.service.registry.RegistryClient.afterConstruct(RegistryClient.java:77)
242 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
243 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
244 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
245 at java.lang.reflect.Method.invoke(Method.java:498)
246 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessorL i f e c y c l e E l e m e n t . i n v o k e ( I n i t D e s t r o y A n n o t a t i o n B e a n P o s t P r o c e s s o r . j a v a : 389 ) 247 a t o r g . s p r i n g f r a m e w o r k . b e a n s . f a c t o r y . a n n o t a t i o n . I n i t D e s t r o y A n n o t a t i o n B e a n P o s t P r o c e s s o r LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) 247 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)247atorg.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessorLifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)248 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
249 … 46 common frames omitted
250 Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /dolphinscheduler/nodes/master
251 at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
252 at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
253 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1450)
254 at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1216)
255 at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1193)
256 at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
257 at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1190)
258 at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:605)
259 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:595)
260 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:48)
261 at org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperRegistry.put(ZookeeperRegistry.java:184)
这里最新版本的ds与zk版本不兼容，最新版本的ds需要最新版本的zk，即3.8版本，当前环境安装的是3.4.6。通过查看ds依赖的zk的jarb包也能验证这一点
解决思路：下载3.8版本的zk并安装
下载地址:
https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
安装方法可以看这篇文章《zookeeper快速入门》
安装完成并启动zk，然后再重新启动ds成功。
访问master节点url连接：
http://<your_ip>:12345/dolphinscheduler/ui/login
默认账户密码：admin/dolphinscheduler123
3 [附加]ds源码修改与打包
当前ds-3.1.3版本数据源还不支持trino，可以修改源码，增加对trino数据源的支持，修改完成后使用如下命令重新编译并完成打包。
./mvnw clean install-Prelease-DskipTests=true
这里的-DskipTests=true指跳过测试，如果不加该参数，整个耗时会特别长。编译完成后可看到如下成功页面。
然后进入你的maven仓库即可发现已经打包的ds-3.1.3安装包
cd /root/.m2/repository/org/apache/dolphinscheduler/dolphinscheduler-dist/3.1.3
ll -h
版权归原作者 不以物喜2020 所有， 如有侵权，请联系我们删除。