0


Apache Doris 安装部署指南

前言

本文隶属于专栏《大数据安装部署》,该专栏为笔者原创,引用请注明来源,不足和错误之处请在评论区帮忙指出,谢谢!


准备

建议下载下面的 2 个安装包

Apache Doris 1.1.2 FE 安装包

Apache Doris 1.1.2 BE 安装包

另外建议参考我的这篇博客部署好 MySQL 客户端——CentOS 7 安装 MySQL 5.7


部署 FE

首先将 FE 安装包复制到指定的节点。

配置 FE

配置文件是

conf/fe.conf

注意:meta_dir表示元数据存储位置,默认值为 ${DORIS_HOME}/doris-meta,目录不存在则需要手动创建(安装包解压后默认是存在的)。

fe.conf 中的JAVA_OPTS默认为java的最大堆内存为4GB,建议将生产环境调整为超过8G。

笔者的 fe.conf 配置如下,各位同学可以参考。注意,一般情况下这个文件用默认的就行了,除非出现端口冲突的情况。

在这里插入图片描述


启动 FE

在 fe 的主目录下执行下面的命令(笔者的 FE 主目录是 /opt/bigdata/doris/apache-doris-fe-1.1.2-bin/fe)

bin/start_fe.sh --daemon

FE 进程启动并进入后台执行。

默认情况下,日志存储在 ${DORIS_HOME}/log 中。

如果启动失败,您可以通过查看 log/fe.log 或 log/fe.out 来查看错误消息。


部署 BE

将 BE 安装包复制到所有节点以部署 BE。

修改所有的 BE 配置

修改

be/conf/be.conf

主要配置

storage_root_path

:数据存储目录。默认值是 be/storage,目录不存在则需要手动创建(安装包解压后默认是存在的)。在多目录的情况下,使用英文分号

;

分离(不要在最后一个目录之后添加

;

)。

笔者的 be.conf 配置如下,各位同学可以参考,注意端口冲突请自行更改(可以使用这个命令

netstat -anp | grep 端口号

查看端口是否被占用)

在这里插入图片描述


将所有 BE 节点添加到 FE

BE 节点需要在 FE 中添加,然后才能加入集群。

建议使用 mysql-client(下载 MySQL 5.7)连接到 FE:

如果要用 Apache Doris 的话,建议还是装个 MySQL,毕竟完美兼容 MySQL 的协议和语法。

建议参考我的这篇博客安装好 MySQL——CentOS 7 安装 MySQL 5.7

./mysql-client -h fe_host -P query_port -uroot
  • fe_host:FE 所在的节点IP;
  • query_port:fe/conf/fe.conf 中的 query_port;
  • u:默认使用 root 帐户,无需输入密码即可登录。

命令执行后,可以看到如下的输出:

[root@node1 ~]# mysql -h node1 -P 9030 -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.7.37 Doris version 1.1.2-rc05-a8323dae4

Copyright (c)2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h'for help. Type '\c' to clear the current input statement.

mysql>

登录后,执行以下命令来添加每个BE:

ALTER SYSTEM ADD BACKEND "be_host:heartbeat_service_port";
  • be_host:BE 所在的节点IP;
  • heartbeat_service_port:be/conf/be.conf 中的 heartbeat_service_port。

如下所示:

mysql>ALTER SYSTEM ADD BACKEND "node1:9050";
Query OK,0rows affected (0.16 sec)
mysql>ALTER SYSTEM ADD BACKEND "node2:9050";
Query OK,0rows affected (0.01 sec)
mysql>ALTER SYSTEM ADD BACKEND "node3:9050";
Query OK,0rows affected (0.00 sec)

启动 BE

在每一个节点的 be 的主目录下执行下面的命令(笔者的 BE 主目录是 /opt/bigdata/doris/apache-doris-be-1.1.2-bin-x86_64/be)

bin/start_be.sh --daemon

BE 进行启动并进入后台执行。

默认情况下,日志存储在 be/log 目录中。

如果启动失败,可以通过查看 be/log/be.log 或 be/log/be.out 来查看错误消息。


至此,实际上安装部署过程已经结束了,但是我们需要检查一下安装的有没有问题,Apache Doris 集群能不能正常工作。

检查

查看 BE 状态

使用 MySQL 客户端连接 FE 并执行下面的命令查看 BE 的状态:

SHOWPROC'/backends';

如果一切正常的话,

Alive

列应该是

true

,如下所示

mysql>SHOWPROC'/backends';+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+----------------------+-------------------------------------------------------------------------------------------------------------------------------+| BackendId | Cluster         | IP            | HostName | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | Tag                      | ErrMsg | Version              |Status|+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+----------------------+-------------------------------------------------------------------------------------------------------------------------------+|10002| default_cluster |192.168.10.11| node1    |9050|9060|8040|8060|2022-09-2609:59:55|2022-09-2610:13:58|true|false|false|0|0.000|12.509 GB     |16.986 GB     |26.36%|26.36%| {"location" : "default"} ||1.1.2-rc05-a8323dae4 | {"lastSuccessReportTabletsTime":"2022-09-26 10:13:44","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} ||10003| default_cluster |192.168.10.12| node2    |9050|9060|8040|8060|2022-09-2610:13:51|2022-09-2610:13:58|true|false|false|0|0.000|14.046 GB     |16.986 GB     |17.31%|17.31%| {"location" : "default"} ||1.1.2-rc05-a8323dae4 | {"lastSuccessReportTabletsTime":"2022-09-26 10:13:55","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} ||10004| default_cluster |192.168.10.13| node3    |9050|9060|8040|8060|2022-09-2610:13:10|2022-09-2610:13:58|true|false|false|0|0.000|14.261 GB     |16.986 GB     |16.04%|16.04%| {"location" : "default"} ||1.1.2-rc05-a8323dae4 | {"lastSuccessReportTabletsTime":"2022-09-26 10:13:15","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} |+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------------+--------+----------------------+-------------------------------------------------------------------------------------------------------------------------------+3rowsinset(0.01 sec)

查看 FE 日志

在 fe 的 log 目录下搜索关键字

thrift server started

,出现如下的输出,说明 FE 是正常工作的。

[root@node1 log]# grep -nR -A 10 "thrift server started" .
./fe.log:36:2022-09-26 09:07:12,649 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [FeServer.start():48] thrift server started.
./fe.log-37-2022-09-26 09:07:12,890 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [RestartApplicationListener.onApplicationStartingEvent():93] Restart disabled due to System property 'spring.devtools.restart.enabled' being set to false
./fe.log-38-2022-09-26 09:07:13,617 INFO (background-preinit|108) [Version.<clinit>():27] HV000001: Hibernate Validator 5.1.0.Final
./fe.log-39-2022-09-26 09:07:14,188 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [StartupInfoLogger.logStarting():55] Starting PaloFe v1.0-SNAPSHOT using Java 1.8.0_261 on node1 with PID 20366 (/opt/bigdata/doris/apache-doris-fe-1.1.2-bin/fe/lib/doris-fe.jar started by root in /opt/bigdata/doris/apache-doris-fe-1.1.2-bin/fe)
./fe.log-40-2022-09-26 09:07:14,194 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [SpringApplication.logStartupProfileInfo():634] No active profile set, falling back to 1 default profile: "default"
./fe.log-41-2022-09-26 09:07:14,310 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [DeferredLog.logTo():255] For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
./fe.log-42-2022-09-26 09:07:15,651 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [RepositoryConfigurationDelegate.registerRepositoriesIn():132] Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
./fe.log-43-2022-09-26 09:07:15,682 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [RepositoryConfigurationDelegate.registerRepositoriesIn():201] Finished Spring Data repository scanning in 19 ms. Found 0 LDAP repository interfaces.
./fe.log-44-2022-09-26 09:07:17,021 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [JettyServletWebServerFactory.getWebServer():166] Server initialized with port: 8030
./fe.log-45-2022-09-26 09:07:17,104 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [ServletWebServerApplicationContext.prepareWebApplicationContext():292] Root WebApplicationContext: initialization completed in 2791 ms
./fe.log-46-2022-09-26 09:07:18,362 INFO (UNKNOWN 192.168.10.11_9010_1664154423601(-1)|1) [WelcomePageHandlerMapping.<init>():53] Adding welcome page: class path resource [static/index.html]
You have new mail in /var/spool/mail/root

Restful API 检查

检查 FE

http://192.168.10.11:8030/api/bootstrap

在这里插入图片描述

检查 BE

http://192.168.10.11:8040/api/health

在这里插入图片描述

http://192.168.10.12:8040/api/health

在这里插入图片描述

http://192.168.10.13:8040/api/health

在这里插入图片描述


MySQL 客户端检查

MySQL 客户端检查 FE

mysql> show frontends\G;
*************************** 1. row ***************************
             Name: 192.168.10.11_9010_1664154423601
               IP: 192.168.10.11
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: true
        ClusterId: 1565676480
             Join: true
            Alive: true
ReplayedJournalId: 1829
    LastHeartbeat: 2022-09-26 10:48:15
         IsHelper: true
           ErrMsg: 
          Version: 1.1.2-rc05-a8323dae4
 CurrentConnected: Yes
1 row inset(0.16 sec)

MySQL 客户端检查 BE

mysql> SHOW BACKENDS\G;
*************************** 1. row ***************************
            BackendId: 10002
              Cluster: default_cluster
                   IP: 192.168.10.11
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2022-09-26 10:37:47
        LastHeartbeat: 2022-09-26 10:49:10
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: 0.000 
        AvailCapacity: 12.503 GB
        TotalCapacity: 16.986 GB
              UsedPct: 26.39 %
       MaxDiskUsedPct: 26.39 %
                  Tag: {"location":"default"}
               ErrMsg: 
              Version: 1.1.2-rc05-a8323dae4
               Status: {"lastSuccessReportTabletsTime":"2022-09-26 10:48:23","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
*************************** 2. row ***************************
            BackendId: 10003
              Cluster: default_cluster
                   IP: 192.168.10.12
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2022-09-26 10:13:51
        LastHeartbeat: 2022-09-26 10:49:10
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: 0.000 
        AvailCapacity: 14.046 GB
        TotalCapacity: 16.986 GB
              UsedPct: 17.31 %
       MaxDiskUsedPct: 17.31 %
                  Tag: {"location":"default"}
               ErrMsg: 
              Version: 1.1.2-rc05-a8323dae4
               Status: {"lastSuccessReportTabletsTime":"2022-09-26 10:48:28","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
*************************** 3. row ***************************
            BackendId: 10004
              Cluster: default_cluster
                   IP: 192.168.10.13
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2022-09-26 10:13:10
        LastHeartbeat: 2022-09-26 10:49:10
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: 0.000 
        AvailCapacity: 14.261 GB
        TotalCapacity: 16.986 GB
              UsedPct: 16.04 %
       MaxDiskUsedPct: 16.04 %
                  Tag: {"location":"default"}
               ErrMsg: 
              Version: 1.1.2-rc05-a8323dae4
               Status: {"lastSuccessReportTabletsTime":"2022-09-26 10:48:55","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}3 rows inset(0.01 sec)

WEB UI

我们使用用户 root 登录 WEB UI,密码为空即可。

在这里插入图片描述

登录后可以看到如下的界面:

可以在 Playground 中执行 SQL 命令操作 Apache Doris 集群。


第一次使用

创建数据库

createdatabase demo;

创建表

use demo;CREATETABLEIFNOTEXISTS demo.example_tbl
(`user_id` LARGEINT NOTNULLCOMMENT"user id",`date`DATENOTNULLCOMMENT"",`city`VARCHAR(20)COMMENT"",`age`SMALLINTCOMMENT"",`sex`TINYINTCOMMENT"",`last_visit_date`DATETIMEREPLACEDEFAULT"1970-01-01 00:00:00"COMMENT"",`cost`BIGINT SUM DEFAULT"0"COMMENT"",`max_dwell_time`INT MAX DEFAULT"0"COMMENT"",`min_dwell_time`INT MIN DEFAULT"99999"COMMENT"")
AGGREGATE KEY(`user_id`,`date`,`city`,`age`,`sex`)DISTRIBUTEDBYHASH(`user_id`) BUCKETS 1
PROPERTIES ("replication_allocation"="tag.location.default: 1");

示例数据

10000,2017-10-01,beijing,20,0,2017-10-01 06:00:00,20,10,10
10006,2017-10-01,beijing,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,beijing,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,shanghai,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,guangzhou,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,shenzhen,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,shenzhen,35,0,2017-10-03 10:20:22,11,6,6

将上述数据保存到 test.csv 文件中。


导入数据

在这里,我们将保存到上述文件中的数据通过 通过

_stream_load

导入我们刚刚创建的表中。

[root@node1 ~]# curl --location-trusted -u root: -T test.csv -H "column_separator:," http://node1:8030/api/demo/example_tbl/_stream_load{"TxnId":2,
    "Label":"c6780496-ac8f-4784-b059-013dec735511",
    "TwoPhaseCommit":"false",
    "Status":"Success",
    "Message":"OK",
    "NumberTotalRows":7,
    "NumberLoadedRows":7,
    "NumberFilteredRows":0,
    "NumberUnselectedRows":0,
    "LoadBytes":411,
    "LoadTimeMs":399,
    "BeginTxnTimeMs":54,
    "StreamLoadPutTimeMs":229,
    "ReadDataTimeMs":0,
    "WriteDataTimeMs":18,
    "CommitAndPublishTimeMs":93}

查询数据

mysql>select * from example_tbl;
+---------+------------+-----------+------+------+---------------------+------+----------------+----------------+
| user_id |date| city      | age  | sex  | last_visit_date     | cost | max_dwell_time | min_dwell_time |
+---------+------------+-----------+------+------+---------------------+------+----------------+----------------+
|10000|2017-10-01 | beijing   |20|0|2017-10-01 06:00:00 |20|10|10||10001|2017-10-01 | beijing   |30|1|2017-10-01 17:05:45 |2|22|22||10002|2017-10-02 | shanghai  |20|1|2017-10-02 12:59:12 |200|5|5||10003|2017-10-02 | guangzhou |32|0|2017-10-02 11:20:00 |30|11|11||10004|2017-10-01 | shenzhen  |35|0|2017-10-01 10:00:15 |100|3|3||10004|2017-10-03 | shenzhen  |35|0|2017-10-03 10:20:22 |11|6|6||10006|2017-10-01 | beijing   |20|0|2017-10-01 07:00:00 |15|2|2|
+---------+------------+-----------+------+------+---------------------+------+----------------+----------------+
7 rows inset(0.17 sec)

mysql>select * from example_tbl where city='shanghai';
+---------+------------+----------+------+------+---------------------+------+----------------+----------------+
| user_id |date| city     | age  | sex  | last_visit_date     | cost | max_dwell_time | min_dwell_time |
+---------+------------+----------+------+------+---------------------+------+----------------+----------------+
|10002|2017-10-02 | shanghai |20|1|2017-10-02 12:59:12 |200|5|5|
+---------+------------+----------+------+------+---------------------+------+----------------+----------------+
1 row inset(0.14 sec)

mysql>select city, sum(cost) as total_cost from example_tbl group by city;
+-----------+------------+
| city      | total_cost |
+-----------+------------+
| beijing   |37|| shenzhen  |111|| guangzhou |30|| shanghai  |200|
+-----------+------------+
4 rows inset(0.16 sec)
标签: Doris 大数据

本文转载自: https://blog.csdn.net/Shockang/article/details/127145991
版权归原作者 Shockang 所有, 如有侵权,请联系我们删除。

“Apache Doris 安装部署指南”的评论:

还没有评论