0
0

CDH 6.3.1 史上最全安装手册

因项目需要CDH，经过十来次的重复安装，反复踩坑、填坑、验证，终于了今日的成功。

基础设施

设置主机

Nodecloudera-scm-servercloudera-scm-agent操作系统cpu内存cdh-guance01✅✅Centos7.44核16Gcdh-guance02❎✅Centos7.44核16Gcdh-guance03❎✅Centos7.44核16G

cloudera-scm-server

一般是一个节点,

cloudera-scm-agent

可以是多个节点

  • 主机名

hostnamectl set-hostname cdh-guance01

hostnamectl set-hostname cdh-guance02

hostnamectl set-hostname cdh-guance03

  • hosts 配置（所有节点）
echo"10.0.16.217 cdh-guance01">> /etc/hosts
echo"10.0.16.218 cdh-guance02">> /etc/hosts
echo"10.0.16.219 cdh-guance03">> /etc/hosts

免密互通（所有节点）

  • 生成密钥

ssh-keygen -t rsa

  • 复制密钥
ssh-copy-id cdh-guance01
ssh-copy-id cdh-guance02
ssh-copy-id cdh-guance03

资源（所有节点）

echo'* soft nproc 65535'>> /etc/security/limits.conf
echo'* hard nproc 65535'>> /etc/security/limits.conf
echo'* soft nofile 65535'>> /etc/security/limits.conf
echo'* hard nofile 65535'>> /etc/security/limits.conf

CDH 资源下载（主节点）

[root@cdh-guance01 opt]# ll -h
总用量 3.9G
drwxr-xr-x 2 root root 4.0K 8月  12 09:57 bin
-rw-r--r-- 1 root root 3.9G 8月  12 09:49 cdh6.3.1-all.zip

安装 MySQL（主节点）

  • 安装

yum install libaio -y

yum -y install autoconf

wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

rpm -ivh mysql57-community-release-el7-11.noarch.rpm

rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022

yum -y install mysql-community-server

systemctl start mysqld

  • 查看初始密码
[root@cdh-guance01 opt]# grep "password" /var/log/mysqld.log2024-08-12T09:23:08.653194Z 1[Note] A temporary password is generated for root@localhost: kWTrBTxyq8/U
  • 设置数据库
[root@cdh-guance01 opt]# mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root:

修改密码后，后面全部输入 

Y

。

  • 修改密码策略（按需，可以不用）
> mysql -uroot-p# 查看当前安全策略
mysql> show variables like 'validate_password%';
+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password_check_user_name    | OFF    || validate_password_dictionary_file    ||| validate_password_length             |8|| validate_password_mixed_case_count   |1|| validate_password_number_count       |1|| validate_password_policy             | MEDIUM || validate_password_special_char_count |1|
+--------------------------------------+--------+
7 rows inset(0.00 sec)# 最低安全策略(0->LOW ,1->MEDIUM, 2->STRONG)
mysql>set global validate_password_policy=0;# 最小密码字符长度是4
mysql>set global validate_password_length=4;# 设置本机root密码
mysql>alter user 'root'@'localhost' identified by 'root';# 刷新权限
mysql>flush privileges;#退出
mysql>exit

mysql 的密码可以设置复杂点，因为我设置的过于简单，结果数据库被莫名其妙的清空了。

  • 关闭 ssl

初始化 scm 的时候一直报数据库异常，ssl有点问题，我被这个问题困了一天，关闭 ssl 提前脱困。

mysql> SHOW VARIABLES LIKE 'have_ssl';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_ssl      | YES   |
+---------------+-------+
1 row inset(0.00 sec)

mysql> SHOW VARIABLES LIKE 'ssl%';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| ssl_ca        | ca.pem          || ssl_capath    ||| ssl_cert      | server-cert.pem || ssl_cipher    ||| ssl_crl       ||| ssl_crlpath   ||| ssl_key       | server-key.pem  |
+---------------+-----------------+
7 rows inset(0.00 sec)

如果 

have_ssl

的值是 

YES

就需要关闭。

找到MySQL的配置文件my.cnf或my.ini，通常位于/etc/mysql/目录下。在[mysqld]区块中添加以下配置来禁用SSL：

[mysqld]
ssl=0

调整后，重启 mysql 服务，在进行查看。

systemctl restart mysqld

查看 ssl。

mysql> SHOW VARIABLES LIKE 'have_ssl';
+---------------+----------+
| Variable_name | Value    |
+---------------+----------+
| have_ssl      | DISABLED |
+---------------+----------+
1 row inset(0.00 sec)
  • 创建scm 帐号和数据库

后面安装集群会用到，在 mysql 命令行依次执行以下命令

set global validate_password_policy=0;
set global validate_password_length=4;
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON *.* TO 'scm'@'%' IDENTIFIED BY 'sQnj2CzMHPSm7BJc';

安装 rsync（所有节点）

用于同步文件，非必选，建议安装，提升安装效率。

yum install -y rsync

JDK（所有节点）

  • 安装

yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y

[root@cdh-guance01 opt]# java -version
openjdk version "1.8.0_412"
OpenJDK Runtime Environment (build 1.8.0_412-b08)
OpenJDK 64-Bit Server VM (build 25.412-b08, mixed mode
  • 配置环境变量

新增环境变量文件 

/etc/profile.d/my_env.sh

，内容如下：

exportJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64
exportCLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
exportPATH=$PATH:$JAVA_HOME/bin
  • 生效

source /etc/profile.d/my_env.sh

CM 安装

下载解压

这里没有提供下载地址，百度网盘下载太慢了，4G 的压缩包，有需要的话，给我留言，通过 QQ 发送。

[root@cdh-guance01 opt]# ll
总用量 4018204
drwxr-xr-x 2 root root       40968月  1314:09 bin
-rw-r--r-- 1 root root 41146033348月  1314:09 cdh6.3.1-all.zip
-rw-r--r-- 1 root root      256804月  272017 mysql57-community-release-el7-11.noarch.rpm
[root@cdh-guance01 opt]# unzip cdh6.3.1-all.zip 
Archive:  cdh6.3.1-all.zip
   creating: cdh6.3.1/
  inflating: cdh6.3.1/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel  
  inflating: cdh6.3.1/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1  
   creating: cdh6.3.1/cm6.3.1/
  inflating: cdh6.3.1/cm6.3.1/cm6.3.1-redhat7.tar.gz  
  inflating: cdh6.3.1/cm6.3.1/jre-8u181-linux-x64.tar.gz  
  inflating: cdh6.3.1/cm6.3.1/mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz  
  inflating: cdh6.3.1/cm6.3.1/mysql-connector-java-5.1.47.jar  
  inflating: cdh6.3.1/manifest.json

分发 mysql-connector-java.jar

CM 是java程序，链接数据库可以使用多种数据库类型，这里主要使用 mysql，需要添加相应的依赖。将 

mysql-connector-java-5.1.27-bin.jar

上传到

/usr/share/java

路径下，并重命名

mysql-connector-java.jar

,分发出去

  • 编写分发程序 xsync
#在该文件中编写如下代码#!/bin/bash#1 获取输入参数个数，如果没有参数，直接退出pcount=$#if((pcount==0));thenecho no args;exit;fi#2 获取文件名称p1=$1fname=`basename $p1`echofname=$fname#3 获取上级目录到绝对路径pdir=`cd-P$(dirname $p1);pwd`echopdir=$pdir#4 获取当前用户名称user=`whoami`#5 循环for((host=1; host<4; host++));doecho ------------------- cdh-guance0$host --------------
        rsync-av$pdir/$fname$user@cdh-guance0$host:$pdirdone
  • 分发
[root@cdh-guance01 opt]# cd cdh6.3.1/cm6.3.1/[root@cdh-guance01 cm6.3.1]# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar[root@cdh-guance01 bin]# ./xsync /usr/share/java/fname=java
pdir=/usr/share
------------------- cdh-guance01 --------------
sending incremental file list

sent 94 bytes  received 17 bytes  74.00 bytes/sec
total size is 1,007,502  speedup is 9,076.59
------------------- cdh-guance02 --------------
sending incremental file list
java/
java/mysql-connector-java.jar

sent 1,007,878 bytes  received 39 bytes  2,015,834.00 bytes/sec
total size is 1,007,502  speedup is 1.00
------------------- cdh-guance03 --------------
sending incremental file list
java/
java/mysql-connector-java.jar

sent 1,007,878 bytes  received 39 bytes  2,015,834.00 bytes/sec
total size is 1,007,502  speedup is 1.00

准备工作

[root@cdh-guance01 opt]# mkdir /opt/cloudera-manager[root@cdh-guance01 opt]# cd cdh6.3.1/cm6.3.1/[root@cdh-guance01 cm6.3.1]# ll
总用量 1993012
-rw-r--r-- 1 root root 14114441476月  162021 cm6.3.1-redhat7.tar.gz
-rw-r--r-- 1 root root   811918936月  162021 jre-8u181-linux-x64.tar.gz
-rw-r--r-- 1 root root  5481936376月  162021 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
[root@cdh-guance01 cm6.3.1]# tar -zxvf cm6.3.1-redhat7.tar.gz ...
[root@cdh-guance01 cm6.3.1]# cd cm6.3.1/RPMS/x86_64/[root@cdh-guance01 x86_64]# mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/[root@cdh-guance01 x86_64]# mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/[root@cdh-guance01 x86_64]# mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/[root@cdh-guance01 x86_64]# cd /opt/cloudera-manager/[root@cdh-guance01 cloudera-manager]# ll
总用量 1185876
-rw-r--r-- 120012001104835689月  252019 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 12001200112038324649月  252019 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r-- 120012001114889月  252019 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
[root@cdh-guance01 cloudera-manager]# cd /opt/bin/# 分发文件到其他机器[root@cdh-guance01 bin]# ./xsync /opt/cloudera-manager/

安装 cloudera-manager-daemons （所有节点）

[root@cdh-guance01 x86_64]# cd /opt/cloudera-manager/[root@cdh-guance01 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

安装 cloudera-manager-agent（所有节点）

yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

调整 server host，修改

/etc/cloudera-scm-agent/config.ini

的

server_host

，设置为

cloudera-manager-server

主机名，如

server_host=cdh-guance01

。

等 

cloudera-manager-server

安装启动成功后，再启动 agent，否则会启动失败。

systemctl start cloudera-scm-agent

安装 cloudera-manager-server（主节点）

  • 安装 cloudera-manager-server
# 安装cloudera-manager-server[root@cdh-guance01 cloudera-manager]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
  • 配置离线仓库
[root@cdh-guance01 cdh6.3.1]# pwd
/opt/cdh6.3.1
[root@cdh-guance01 cdh6.3.1]# ll
总用量 2035088
-rw-r--r-- 1 root root 2083878000 6月  16 2021 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
-rw-r--r-- 1 root root         40 6月  16 2021 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
drwxr-xr-x 3 root root       4096 8月  12 19:26 cm6.3.1
-rw-r--r-- 1 root root      33887 6月  16 2021 manifest.json
[root@cdh-guance01 cdh6.3.1]# mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel* /opt/cloudera/parcel-repo
[root@cdh-guance01 cdh6.3.1]# mv manifest.json /opt/cloudera/parcel-repo
[root@cdh-guance01 cdh6.3.1]# cd /opt/cloudera/parcel-repo
[root@cdh-guance01 parcel-repo]# ll
总用量 2035084
-rw-r--r-- 1 root root 2083878000 6月  16 2021 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
-rw-r--r-- 1 root root         40 6月  16 2021 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
-rw-r--r-- 1 root root      33887 6月  16 2021 manifest.json
[root@cdh-guance01 parcel-repo]# mv CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
  • 调整 DB 配置
/etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdh-guance01:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=root
com.cloudera.cmf.db.setupType=External
  • 启动 cloudera-scm-server
# 初始化数据库[root@cdh-guance01 parcel-repo]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password: 
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration filein /etc/cloudera-scm-server
Executing:  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Mon Aug 1220:08:42 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or setuseSSL=true and provide truststore for server certificate verification.
[2024-08-12 20:08:42,790] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!# 启动服务[root@cdh-guance01 parcel-repo]# systemctl start cloudera-scm-server

登陆配置 CDH 组件

http://cdh-guance01:7180，帐号信息 admin/admin

在这里插入图片描述
在这里插入图片描述在这里插入图片描述
在这里插入图片描述

在这里插入图片描述

  • 组件安装，按实际需求选择对应的组件在这里插入图片描述在这里插入图片描述在这里插入图片描述
  • 数据库配置

为了简便操作，我直接授权 scm 所有的 databases 权限，弱密码有风险，建议设置复杂点。
在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

常见问题

NFS Gateway 启动异常
在这里插入图片描述

解决方案：启动 

rpcbind

服务

[root@cdh-guance02 ~]# service rpcbind start
Redirecting to /bin/systemctl start rpcbind.service
[root@cdh-guance02 ~]# systemctl status rpcbind
● rpcbind.service - RPC bindservice
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since 二 2024-08-13 09:19:23 CST; 5s ago
  Process: 30123ExecStart=/sbin/rpcbind -w$RPCBIND_ARGS(code=exited, status=0/SUCCESS)
 Main PID: 30124(rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─30124 /sbin/rpcbind -w8月 13 09:19:23 cdh-guance02 systemd[1]: Starting RPC bind service...
8月 13 09:19:23 cdh-guance02 systemd[1]: Started RPC bind service.

MainThread agent ERROR Error, CM server guid updated

[14/Aug/2024 14:01:30 +0000]29251 MainThread agent        ERROR    Error, CM server guid updated, expected 2dab60bc-d941-438b-8454-c7cb271f4ad6, received cd48128c-8c13-4b96-ab8f-4b8cc20008d4
[14/Aug/2024 14:01:30 +0000]29251 MainThread agent        ERROR    Error, CM server guid updated, expected 2dab60bc-d941-438b-8454-c7cb271f4ad6, received cd48128c-8c13-4b96-ab8f-4b8cc20008d4
[14/Aug/2024 14:01:34 +0000]29251 MainThread agent        ERROR    Error, CM server guid updated, expected 2dab60bc-d941-438b-8454-c7cb271f4ad6, received cd48128c-8c13-4b96-ab8f-4b8cc20008d4
[14/Aug/2024 14:01:34 +0000]29251 MainThread agent        ERROR    Error, CM server guid updated, expected 2dab60bc-d941-438b-8454-c7cb271f4ad6, received cd48128c-8c13-4b96-ab8f-4b8cc20008d4
[14/Aug/2024 14:01:39 +0000]29251 MainThread agent        ERROR    Error, CM server guid updated, expected 2dab60bc-d941-438b-8454-c7cb271f4ad6, received cd48128c-8c13-4b96-ab8f-4b8cc20008d4

在这里插入图片描述
解决方案：

删除 cm_guid

sudo rm /var/lib/cloudera-scm-agent/cm_guid

重启 cloudera-scm-agent

systemctl restart cloudera-scm-agent

标签： 大数据 运维
本文转载自: https://blog.csdn.net/liurui_wuhan/article/details/141201498
版权归原作者 A心有千千结 所有， 如有侵权，请联系我们删除。
登录后发布评论

“CDH 6.3.1 史上最全安装手册”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

***安全登录注册系统全解析

Ubuntu启用ROOT用户

selenium+谷歌驱动器的安装

VSCode的下载安装前期准备

分享 11 个常用的 Nginx 性能优化参数工作

git记住账号和密码

大数据新视界 --大数据大厂之 Presto 性能优化秘籍：加速大数据交互式查询

文章导航