0


PySpark完美安装

一、hadoop版本号确认

  1. hadoop ==> 2.7.2

[root@dm46 TDH-Client]# hadoop version

Hadoop 2.7.2-transwarp-6.2.0

Subversion http://xxxx:10080/hadoop/hadoop-2.7.2-transwarp.git -r f31230971c2a36e77e4886e0f621366826cec3a3

Compiled by jenkins on 2019-07-27T11:33Z

Compiled with protoc 2.5.0

二、下载spark

注意:选择 Pre-built for Apache Hadoop 2.7

https://dlcdn.apache.org/spark/spark-3.2.3/spark-3.2.3-bin-hadoop2.7.tgz

三、准备pyspark环境

  1. 创建虚拟环境

conda create -n pyspark_env python=3.8

conda activate pyspark_env

  1. 安装pyspark

注意:确保spark版本与pyspark版本一致

pip install pyspark==3.2.3 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

3.测试

cd /home/xxx/kdh/spark/bin

./spark-submit --master local[*] /home/xxx/kdh/spark/examples/src/main/python/pi.py 30

四、常见报错

  1. RuntimeError: Python in worker has different version 3.8 than that in driver 3.6

原因:local模式下,此时使用的是当前环境的python。如果代码里指定了pyspark_env,当前环境也应该是pyspark_env。

conda activate pyspark_env

cd /home/xxx/kdh/spark/bin

./spark-submit --master local[*] /home/xxx/workspace/pyspark_learn/02_pyspark_core/main/02_Wordcount_hdfs.py

五、其它

  1. arrow安装

pip install pyspark[sql] -i https://pypi.tuna.tsinghua.edu.cn/simple

标签: spark

本文转载自: https://blog.csdn.net/aidway/article/details/129541674
版权归原作者 kangwq2017 所有, 如有侵权,请联系我们删除。

“PySpark完美安装”的评论:

还没有评论