0


pyflink 安装和测试

FPY Warning!
在这里插入图片描述

安装 apache-Flink

# pip install apache-Flink -i https://pypi.tuna.tsinghua.edu.cn/simple/
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/
Collecting apache-Flink
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7f/a3/ad50270f0f9b7738922a170c47ec18061224a5afa0864e8749d27b1d5501/apache_flink-1.20.0-cp38-cp38-manylinux1_x86_64.whl (6.8 MB)|████████████████████████████████|6.8 MB 2.3 MB/s eta 0:00:01
Collecting avro-python3!=1.9.2,>=1.8.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/cc/97/7a6970380ca8db9139a3cc0b0e3e0dd3e4bc584fb3644e1d06e71e1a55f0/avro-python3-1.10.2.tar.gz (38 kB)
Collecting ruamel.yaml>=0.18.4
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/73/67/8ece580cc363331d9a53055130f86b096bf16e38156e33b1d3014fffda6b/ruamel.yaml-0.18.6-py3-none-any.whl (117 kB)|████████████████████████████████|117 kB 10.1 MB/s eta 0:00:01
Requirement already satisfied: python-dateutil<3,>=2.8.0 in /usr/local/lib/python3.8/dist-packages (from apache-Flink)(2.8.2)
Collecting numpy>=1.22.4
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/98/5d/5738903efe0ecb73e51eb44feafba32bdba2081263d40c5043568ff60faf/numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)|████████████████████████████████|17.3 MB 2.3 MB/s eta 0:00:01
Requirement already satisfied: pytz>=2018.3in /usr/local/lib/python3.8/dist-packages (from apache-Flink)(2022.2.1)
Collecting pemja==0.4.1; platform_system !="Windows"
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/23/4c/422c99d5d2b823309888714185dc4467a935a25fb1e265c604855d0ea4b9/pemja-0.4.1-cp38-cp38-manylinux1_x86_64.whl (325 kB)|████████████████████████████████|325 kB 9.8 MB/s eta 0:00:01
Collecting py4j==0.10.9.7
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/10/30/a58b32568f1623aaad7db22aa9eafc4c6c194b429ff35bdc55ca2726da47/py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)|████████████████████████████████|200 kB 10.0 MB/s eta 0:00:01
Collecting protobuf>=3.19.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c0/be/bac52549cab1aaab112d380b3f2a80a348ba7083a80bf4ff4be4fb5a6729/protobuf-5.28.1-cp38-abi3-manylinux2014_x86_64.whl (316 kB)|████████████████████████████████|316 kB 10.0 MB/s eta 0:00:01
Collecting apache-beam<2.49.0,>=2.43.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9d/1b/59d9717241170b707fbdd82fa74a676260e7fa03fecfa7fafd58f0c178e1/apache_beam-2.48.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.8 MB)|████████████████████████████████|14.8 MB 2.2 MB/s eta 0:00:01
Collecting requests>=2.26.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)|████████████████████████████████|64 kB 8.1 MB/s  eta 0:00:01
Collecting cloudpickle>=2.2.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/96/43/dae06432d0c4b1dc9e9149ad37b4ca8384cf6eb7700cd9215b177b914f0a/cloudpickle-3.0.0-py3-none-any.whl (20 kB)
Collecting apache-flink-libraries<1.20.1,>=1.20.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1f/31/80cfa34b6a53a5c19d66faf3e7d29cb0689a69c128a381e584863eb9669a/apache-flink-libraries-1.20.0.tar.gz (231.5 MB)|████████████████████████████████|231.5 MB 84 kB/s  eta 0:00:01
Collecting fastavro!=1.8.0,>=1.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3a/6a/1998c619c6e59a35d0a5df49681f0cdfb6dbbeabb0d24d26e938143c655f/fastavro-1.9.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)|████████████████████████████████|3.2 MB 450 kB/s eta 0:00:01
Collecting pyarrow>=5.0.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3f/08/bc497130789833de09e345e3ce4647e3ce86517c4f70f2144f0367ca378b/pyarrow-17.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (40.0 MB)|████████████████████████████████|40.0 MB 1.3 MB/s eta 0:00:01
Collecting pandas>=1.3.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f8/7f/5b047effafbdd34e52c9e2d7e44f729a0655efafb22198c45cf692cdc157/pandas-2.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)|████████████████████████████████|12.4 MB 1.2 MB/s eta 0:00:01
Collecting httplib2>=0.19.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a8/6c/d2fbdaaa5959339d53ba38e94c123e4e84b8fbc4b84beb0e70d7c1608486/httplib2-0.22.0-py3-none-any.whl (96 kB)|████████████████████████████████|96 kB 2.9 MB/s eta 0:00:011
Collecting ruamel.yaml.clib>=0.2.7; platform_python_implementation =="CPython" and python_version <"3.13"
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/22/fa/b2a8fd49c92693e9b9b6b11eef4c2a8aedaca2b521ab3e020aa4778efc23/ruamel.yaml.clib-0.2.8-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (596 kB)|████████████████████████████████|596 kB 2.6 MB/s eta 0:00:01
Requirement already satisfied: six>=1.5in /usr/lib/python3/dist-packages (from python-dateutil<3,>=2.8.0->apache-Flink)(1.14.0)
Collecting find-libpython
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1d/89/6b4624122d5c61a86e8aebcebd377866338b705ce4f115c45b046dc09b99/find_libpython-0.4.0-py3-none-any.whl (8.7 kB)
Collecting proto-plus<2,>=1.7.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7c/6f/db31f0711c0402aa477257205ce7d29e86a75cb52cd19f7afb585f75cda0/proto_plus-1.24.0-py3-none-any.whl (50 kB)|████████████████████████████████|50 kB 3.5 MB/s eta 0:00:011
Collecting grpcio!=1.48.0,<2,>=1.33.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/44/f8975d2719dbf58d4a036f936b6c2adeddc7d2a10c2f7ca6ea87ab4c5086/grpcio-1.66.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB)|████████████████████████████████|5.8 MB 1.3 MB/s eta 0:00:01
Collecting fasteners<1.0,>=0.3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/61/bf/fd60001b3abc5222d8eaa4a204cd8c0ae78e75adc688f33ce4bf25b7fafa/fasteners-0.19-py3-none-any.whl (18 kB)
Collecting crcmod<2.0,>=1.7
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6b/b0/e595ce2a2527e169c3bcd6c33d2473c1918e0b7f6826a043ca1245dd4e5b/crcmod-1.7.tar.gz (89 kB)|████████████████████████████████|89 kB 5.2 MB/s eta 0:00:011
Requirement already satisfied: typing-extensions>=3.7.0 in /usr/local/lib/python3.8/dist-packages (from apache-beam<2.49.0,>=2.43.0->apache-Flink)(4.3.0)
Collecting pydot<2,>=1.2.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ea/76/75b1bb82e9bad3e3d656556eaa353d8cd17c4254393b08ec9786ac8ed273/pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: regex>=2020.6.8 in /usr/local/lib/python3.8/dist-packages (from apache-beam<2.49.0,>=2.43.0->apache-Flink)(2022.10.31)
Collecting hdfs<3.0.0,>=2.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/29/c7/1be559eb10cb7cac0d26373f18656c8037553619ddd4098e50b04ea8b4ab/hdfs-2.7.3.tar.gz (43 kB)|████████████████████████████████|43 kB 5.2 MB/s eta 0:00:011
Collecting orjson<4.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/25/13/a66f4873ed57832aab57dd8b49c91c4c22b35fb1fa0d1dce3bf8928f2fe0/orjson-3.10.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)|████████████████████████████████|141 kB 4.8 MB/s eta 0:00:01
Collecting objsize<0.7.0,>=0.6.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ab/37/e5765c22a491e1cd23fb83059f73e478a2c45a464b2d61c98ef5a8d0681c/objsize-0.6.1-py3-none-any.whl (9.3 kB)
Collecting zstandard<1,>=0.18.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1c/4b/be9f3f9ed33ff4d5e578cf167c16ac1d8542232d5e4831c49b615b5918a6/zstandard-0.23.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)|████████████████████████████████|5.4 MB 1.3 MB/s eta 0:00:01
Collecting pymongo<5.0.0,>=3.8.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2a/72/77445354da27437534ee674faf55a2ef4bfc6ed9b28cbe743d6e7e4c2c61/pymongo-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (685 kB)|████████████████████████████████|685 kB 5.0 MB/s eta 0:00:01
Collecting dill<0.3.2,>=0.3.1.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c7/11/345f3173809cea7f1a193bfbf02403fff250a3360e0e118a1630985e547d/dill-0.3.1.1.tar.gz (151 kB)|████████████████████████████████|151 kB 4.9 MB/s eta 0:00:01
Requirement already satisfied: idna<4,>=2.5in /usr/lib/python3/dist-packages (from requests>=2.26.0->apache-Flink)(2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests>=2.26.0->apache-Flink)(1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->apache-Flink)(2019.11.28)
Collecting charset-normalizer<4,>=2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3d/09/d82fe4a34c5f0585f9ea1df090e2a71eb9bb1e469723053e1ee9f57c16f3/charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)|████████████████████████████████|141 kB 4.9 MB/s eta 0:00:01
Collecting tzdata>=2022.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/65/58/f9c9e6be752e9fcb8b6a0ee9fb87e6e7a1f6bcab2cdc73f02bb7ba91ada0/tzdata-2024.1-py2.py3-none-any.whl (345 kB)|████████████████████████████████|345 kB 4.9 MB/s eta 0:00:01
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version >"3.0"in /usr/local/lib/python3.8/dist-packages (from httplib2>=0.19.0->apache-Flink)(3.0.9)
Collecting docopt
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz (25 kB)
Collecting dnspython<3.0.0,>=1.16.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/87/a1/8c5287991ddb8d3e4662f71356d9656d91ab3a36618c3dd11b280df0d255/dnspython-2.6.1-py3-none-any.whl (307 kB)|████████████████████████████████|307 kB 4.9 MB/s eta 0:00:01
Building wheels for collected packages: avro-python3, apache-flink-libraries, crcmod, hdfs, dill, docopt
  Building wheel for avro-python3 (setup.py)... done
  Created wheel for avro-python3: filename=avro_python3-1.10.2-py3-none-any.whl size=44009sha256=b7630bedfef2c8bd38772ef6e0f6f62520cfc6ee7fbc554f7b1b7ae64eb2229e
  Stored in directory: /root/.cache/pip/wheels/6b/28/4f/3f68740c0cd12549e2ba9fcfad15913841346bc927af36903e
  Building wheel for apache-flink-libraries (setup.py)... done
  Created wheel for apache-flink-libraries: filename=apache_flink_libraries-1.20.0-py2.py3-none-any.whl size=231628008sha256=2a433e291e2192fa73efb6780bb4753f8c9ecddf7a33036afd02583a42bbc424
  Stored in directory: /root/.cache/pip/wheels/1d/6c/ea/da3305a119e44581bd4c0c0e92a376fd0615426227a071894d
  Building wheel for crcmod (setup.py)... done
  Created wheel for crcmod: filename=crcmod-1.7-cp38-cp38-linux_x86_64.whl size=35981sha256=60feb55c2aec53cb53b2c76013f7b37680467cb7c24e3e9e63e2ddcca37960c7
  Stored in directory: /root/.cache/pip/wheels/ee/bf/82/ac509f3b383e310a168c1da020cbc62d98c03a6c7c74babc63
  Building wheel for hdfs (setup.py)... done
  Created wheel for hdfs: filename=hdfs-2.7.3-py3-none-any.whl size=34321sha256=52a3f2b630c182293adaa98f1d54c73d6543155a6c8b536b723ba81bcd2c0c7c
  Stored in directory: /root/.cache/pip/wheels/71/d9/87/dc2129ee8e18b4b82cfc6be6dbba6f5b7091a45cfa53b5855d
  Building wheel for dill (setup.py)... done
  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78530sha256=8cb55478bc96ba4978d75fcc54972b21f8334d990691267a53e684c2c89eb1e9
  Stored in directory: /root/.cache/pip/wheels/8b/d5/ad/893b2f2db5de6f4c281a9d16abeb0618f5249620f845a857b4
  Building wheel for docopt (setup.py)... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704sha256=bf6de70865de15338a3da5fb54a47de343b677743fa989a2289bbf6557330136
  Stored in directory: /root/.cache/pip/wheels/93/51/4b/6e0f7cba524fbe1e9e973f4bc9be5f3ab1e38346d7d63505f4
Successfully built avro-python3 apache-flink-libraries crcmod hdfs dill docopt
ERROR: apache-beam 2.48.0 has requirement cloudpickle~=2.2.1, but you'll have cloudpickle 3.0.0 which is incompatible.
ERROR: apache-beam 2.48.0 has requirement protobuf<4.24.0,>=3.20.3, but you'll have protobuf 5.28.1 which is incompatible.
ERROR: apache-beam 2.48.0 has requirement pyarrow<12.0.0,>=3.0.0, but you'll have pyarrow 17.0.0 which is incompatible.
Installing collected packages: avro-python3, ruamel.yaml.clib, ruamel.yaml, numpy, find-libpython, pemja, py4j, protobuf, cloudpickle, proto-plus, grpcio, fastavro, fasteners, crcmod, charset-normalizer, requests, pyarrow, pydot, docopt, hdfs, orjson, objsize, zstandard, dnspython, pymongo, httplib2, dill, apache-beam, apache-flink-libraries, tzdata, pandas, apache-Flink
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.6
    Uninstalling numpy-1.21.6:
      Successfully uninstalled numpy-1.21.6
  Attempting uninstall: requests
    Found existing installation: requests 2.22.0
    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'requests'. No files were found to uninstall.
  Attempting uninstall: dill
    Found existing installation: dill 0.3.7
    Uninstalling dill-0.3.7:
      Successfully uninstalled dill-0.3.7
  Attempting uninstall: pandas
    Found existing installation: pandas 1.2.0
    Uninstalling pandas-1.2.0:
      Successfully uninstalled pandas-1.2.0
Successfully installed apache-Flink-1.20.0 apache-beam-2.48.0 apache-flink-libraries-1.20.0 avro-python3-1.10.2 charset-normalizer-3.3.2 cloudpickle-3.0.0 crcmod-1.7 dill-0.3.1.1 dnspython-2.6.1 docopt-0.6.2 fastavro-1.9.7 fasteners-0.19 find-libpython-0.4.0 grpcio-1.66.1 hdfs-2.7.3 httplib2-0.22.0 numpy-1.24.4 objsize-0.6.1 orjson-3.10.7 pandas-2.0.3 pemja-0.4.1 proto-plus-1.24.0 protobuf-5.28.1 py4j-0.10.9.7 pyarrow-17.0.0 pydot-1.4.2 pymongo-4.8.0 requests-2.32.3 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 tzdata-2024.1 zstandard-0.23.0

pip 安装后自动会把 flink 也装上

# find / -name flink 2>/dev/null
/usr/local/lib/python3.8/dist-packages/apache_beam/examples/flink
/usr/local/lib/python3.8/dist-packages/apache_beam/io/flink
/usr/local/lib/python3.8/dist-packages/pyflink/bin/flink
/usr/local/lib/python3.8/dist-packages/pyflink/bin/flink

就是 flink 可执行文件

# /usr/local/lib/python3.8/dist-packages/pyflink/bin/flink run -h

Action "run" compiles and runs a program.

  Syntax: run [OPTIONS]<jar-file><arguments>"run" action options:
     -c,--class <classname>                     Class with the program entry
                                                point ("main()" method). Only
                                                needed if the JAR file does not
                                                specify the class in its
                                                manifest.
     -C,--classpath <url>                       Adds a URL to each user code
                                                classloader  on all nodes in the
                                                cluster. The paths must specify
                                                a protocol (e.g. file://) and be
                                                accessible on all nodes (e.g. by
                                                means of a NFS share). You can
                                                use this option multiple timesfor specifying more than one
                                                URL. The protocol must be
                                                supported by the {@link
                                                java.net.URLClassLoader}.
     -cm,--claimMode <arg>                      Defines how should we restore
                                                from the given savepoint.
                                                Supported options: [claim -
                                                claim ownership of the savepoint
                                                and delete once it is subsumed,
                                                no_claim (default) - do not
                                                claim ownership, the first
                                                checkpoint will not reuse any
                                                files from the restored one,
                                                legacy (deprecated) - the old
                                                behaviour, do not assume
                                                ownership of the savepoint
                                                files, but can reuse some shared
                                                files.
     -d,--detached                              If present, runs the job in
                                                detached mode
     -n,--allowNonRestoredState                 Allow to skip savepoint state
                                                that cannot be restored. You
                                                need to allow this if you
                                                removed an operator from your
                                                program that was part of the
                                                program when the savepoint was
                                                triggered.
     -p,--parallelism <parallelism>             The parallelism with which to
                                                run the program. Optional flag
                                                to override the default value
                                                specified in the configuration.
     -py,--python <pythonFile>                  Python script with the program
                                                entry point. The dependent
                                                resources can be configured with
                                                the `--pyFiles` option.
     -pyarch,--pyArchives <arg>                 Add python archive files for
                                                job. The archive files will be
                                                extracted to the working
                                                directory of python UDF worker.
                                                For each archive file, a target
                                                directory be specified. If the
                                                target directory name is
                                                specified, the archive file will
                                                be extracted to a directory with
                                                the specified name. Otherwise,
                                                the archive file will be
                                                extracted to a directory with
                                                the same name of the archive
                                                file. The files uploaded via
                                                this option are accessible via
                                                relative path. '#' could be used
                                                as the separator of the archive
                                                file path and the target
                                                directory name. Comma (',')
                                                could be used as the separator
                                                to specify multiple archive
                                                files. This option can be used
                                                to upload the virtual
                                                environment, the data files used
                                                in Python UDF (e.g.,
                                                --pyArchives
                                                file:///tmp/py37.zip,file:///tmp
                                                /data.zip#data --pyExecutable
                                                py37.zip/py37/bin/python). The
                                                data files could be accessed in
                                                Python UDF, e.g.: f =
                                                open('data/data.txt', 'r').
     -pyclientexec,--pyClientExecutable <arg>   The path of the Python
                                                interpreter used to launch the
                                                Python process when submitting
                                                the Python jobs via "flink run"
                                                or compiling the Java/Scala jobs
                                                containing Python UDFs.
     -pyexec,--pyExecutable <arg>               Specify the path of the python
                                                interpreter used to execute the
                                                python UDF worker (e.g.:
                                                --pyExecutable
                                                /usr/local/bin/python3). The
                                                python UDF worker depends on
                                                Python 3.8+, Apache Beam
                                                (version ==2.43.0), Pip
                                                (version >=20.3) and SetupTools
                                                (version >=37.0.0). Please
                                                ensure that the specified
                                                environment meets the above
                                                requirements.
     -pyfs,--pyFiles <pythonFiles>              Attach custom files for job. The
                                                standard resource file suffixes
                                                such as .py/.egg/.zip/.whl or
                                                directory are all supported.
                                                These files will be added to the
                                                PYTHONPATH of both the local
                                                client and the remote python UDF
                                                worker. Files suffixed with .zip
                                                will be extracted and added to
                                                PYTHONPATH. Comma (',') could be
                                                used as the separator to specify
                                                multiple files (e.g., --pyFiles
                                                file:///tmp/myresource.zip,hdfs:
                                                ///$namenode_address/myresource2
                                                .zip).
     -pym,--pyModule <pythonModule>             Python module with the program
                                                entry point. This option must be
                                                used in conjunction with
                                                `--pyFiles`.
     -pypath,--pyPythonPath <arg>               Specify the path of the python
                                                installation in worker
                                                nodes.(e.g.: --pyPythonPath
                                                /python/lib64/python3.7/).User
                                                can specify multiple paths using
                                                the separator ":".
     -pyreq,--pyRequirements <arg>              Specify a requirements.txt filewhich defines the third-party
                                                dependencies. These dependencies
                                                will be installed and added to
                                                the PYTHONPATH of the python UDF
                                                worker. A directory which
                                                contains the installation
                                                packages of these dependencies
                                                could be specified optionally.
                                                Use '#' as the separator if the
                                                optional parameter exists (e.g.,
                                                --pyRequirements
                                                file:///tmp/requirements.txt#fil
                                                e:///tmp/cached_dir).
     -rm,--restoreMode <arg>                    This option is deprecated,
                                                please use 'claimMode' instead.
     -s,--fromSavepoint <savepointPath>         Path to a savepoint to restore
                                                the job from (for example
                                                hdfs:///flink/savepoint-1537).
     -sae,--shutdownOnAttachedExit              If the job is submitted in
                                                attached mode, perform a
                                                best-effort cluster shutdown
                                                when the CLI is terminated
                                                abruptly, e.g., in response to a
                                                user interrupt, such as typing
                                                Ctrl + C.
  Options for Generic CLI mode:
     -D<property=value>   Allows specifying multiple generic configuration
                           options. The available options can be found at
                           https://nightlies.apache.org/flink/flink-docs-stable/
                           ops/config.html
     -e,--executor <arg>   DEPRECATED: Please use the -t option instead which is
                           also available with the "Application Mode".
                           The name of the executor to be used for executing the
                           given job, which is equivalent to the
                           "execution.target" config option. The currently
                           available executors are: "remote", "local",
                           "kubernetes-session", "yarn-per-job"(deprecated),
                           "yarn-session".
     -t,--target <arg>     The deployment target for the given application,
                           which is equivalent to the "execution.target" config
                           option. For the "run" action the currently available
                           targets are: "remote", "local", "kubernetes-session",
                           "yarn-per-job"(deprecated), "yarn-session". For the
                           "run-application" action the currently available
                           targets are: "kubernetes-application".

  Options for yarn-cluster mode:
     -m,--jobmanager <arg>            Set to yarn-cluster to use YARN execution
                                      mode.
     -yid,--yarnapplicationId <arg>   Attach to running YARN session
     -z,--zookeeperNamespace <arg>    Namespace to create the Zookeeper
                                      sub-paths for high availability mode

  Options for default mode:
     -D<property=value>             Allows specifying multiple generic
                                     configuration options. The available
                                     options can be found at
                                     https://nightlies.apache.org/flink/flink-do
                                     cs-stable/ops/config.html
     -m,--jobmanager <arg>           Address of the JobManager to which to
                                     connect. Use this flag to connect to a
                                     different JobManager than the one specified
                                     in the configuration. Attention: This
                                     option is respected only if the
                                     high-availability configuration is NONE.
     -z,--zookeeperNamespace <arg>   Namespace to create the Zookeeper sub-paths
                                     for high availability mode

flink run 发送作业

!cd /usr/local/lib/python3.8/dist-packages/pyflink/ && ./bin/flink run \
      --jobmanager <<jobmanagerHost>>:8081 \
      --python examples/table/word_count.py

如果你的执行环境没有python或者安装的是python3,会报错:

Cannot run program "python": error=2, No such file or directory
org.apache.flink.client.program.ProgramAbortException: java.io.IOException: Cannot run program "python": error=2, No such file or directory
    at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
    at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
    at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
    at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: java.io.IOException: Cannot run program "python": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378)
    at org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492)
    at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
    ... 14 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 17 more

略施小计,创建软链接指向现有python3即可

!ln -s /usr/bin/python3 /usr/bin/python

在此执行,job被提交到flink集群
在这里插入图片描述
在这里插入图片描述

标签: pyflink flink

本文转载自: https://blog.csdn.net/itnerd/article/details/142181115
版权归原作者 颹蕭蕭 所有, 如有侵权,请联系我们删除。

“pyflink 安装和测试”的评论:

还没有评论