流计算开发平台StreamPark安装

一、StreamPark简介

StreamPark（原StreamX）是一个流处理应用程序开发管理框架。初衷是让流处理更简单，旨在轻松构建和管理流处理应用程序，提供使用 Apache Flink 和 Apache Spark 编写流处理应用程序的开发框架，未来将支持更多其他引擎。同时，StreamPark 提供了一个流处理应用管理平台，核心能力包括但不限于应用开发、调试、交互查询、部署、运维、实时数仓等。

官方网站：https://streampark.apache.org

二、快速安装使用

1. 使用docker-compose安装

1.1 安装docker

首先要安装docker，要求如下：

Docker 1.13.1+
Docker Compose 1.28.0+

1.2 安装docker-compose

docker-compose安装方式有很多，这里在尝试pip安装后，启动StreamPark后频繁报错，所以推荐二进制安装方式。

• 首先下载docker-compose文件，使用daocloud下载会快点，安装最新稳定版v2.16.0sudocurl-L https://get.daocloud.io/docker/compose/releases/download/2.16.0/docker-compose-`uname-s`-`uname-m`-o /usr/local/bin/docker-compose
• 接着将docker-compose添加可执行权限sudochmod +x /usr/local/bin/docker-compose
• 查询版本docker-compose--version

###1.3 下载docker-compose.yaml和.env

官方给的指令：

wget https://github.com/apache/incubator-streampark/blob/dev/deploy/docker/docker-compose.yaml
wget https://github.com/apache/incubator-streampark/blob/dev/deploy/docker/.env

这里我下载到的文件里面是html代码，且下载麻烦，下面直接给文件内容，创建两个文件就行：

sudomkdir-p streampark
sudovim docker-compose.yaml

• docker-compose.yaml

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.version:'3.8'services:streampark-console:image: apache/streampark:latest
    command: ${RUN_COMMAND}ports:- 10000:10000env_file: .env
    volumes:- flink:/streampark/flink/${FLINK}- /var/run/docker.sock:/var/run/docker.sock
      - /etc/hosts:/etc/hosts:ro
      - ~/.kube:/root/.kube:ro
    privileged:truerestart: unless-stopped
    networks:- streampark

  flink-jobmanager:image: ${FLINK_IMAGE}ports:-"8081:8081"command: jobmanager
    volumes:- flink:/opt/flink
    env_file: .env
    restart: unless-stopped
    privileged:truenetworks:- streampark

  flink-taskmanager:image: ${FLINK_IMAGE}depends_on:- flink-jobmanager
    command: taskmanager
    deploy:replicas:1env_file: .env
    restart: unless-stopped
    privileged:truenetworks:- streampark

networks:streampark:driver: bridge

volumes:flink:

这里要注意按照自己的需求调整配置，特别注意端口不能和现有服务端口冲突

• .env

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##   http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing,# software distributed under the License is distributed on an# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY# KIND, either express or implied.  See the License for the# specific language governing permissions and limitations# under the License.#TZ=Asia/Shanghai

SPRING_PROFILES_ACTIVE=h2 #mysql, pgsql# If use mysql or pgsql, please set the following parameters#SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/streampark?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8#SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/streampark?stringtype=unspecified#SPRING_DATASOURCE_USERNAME=root#SPRING_DATASOURCE_PASSWORD=streamparkFLINK=flink1.14.5
FLINK_IMAGE=flink:1.14.5-scala_2.12

RUN_COMMAND='/bin/sh -c "wget -P lib https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.0.31/mysql-connector-j-8.0.31.jar && bash bin/startup.sh"'JOB_MANAGER_RPC_ADDRESS=flink-jobmanager

1.4 启动容器

docker-compose up -d

1.5 查看服务

• 访问StreamPark http://yourip:10000，这里的yourip是指宿主机的ip - 默认账号和密码是admin 和 streampark
• 访问StreamPark http://yourip:8081，同上
• 这里的端口是和配置文件中的一致的，如果有改动，改成对应的端口即可

1.6 快速开始

请见官方文档：[StreamPark-快速开始](https://streampark.apache.org/zh-CN/docs/user-guide/quick-start)

2. 基于源码构建镜像进行StreamPark部署

未完待续。

三、常见报错

未完待续。。。

报错记录不在，后续有空更新