一、hadoop版本号确认
- hadoop ==> 2.7.2
[root@dm46 TDH-Client]# hadoop version
Hadoop 2.7.2-transwarp-6.2.0
Subversion http://xxxx:10080/hadoop/hadoop-2.7.2-transwarp.git -r f31230971c2a36e77e4886e0f621366826cec3a3
Compiled by jenkins on 2019-07-27T11:33Z
Compiled with protoc 2.5.0
二、下载spark
注意:选择 Pre-built for Apache Hadoop 2.7
https://dlcdn.apache.org/spark/spark-3.2.3/spark-3.2.3-bin-hadoop2.7.tgz
三、准备pyspark环境
- 创建虚拟环境
conda create -n pyspark_env python=3.8
conda activate pyspark_env
- 安装pyspark
注意:确保spark版本与pyspark版本一致
pip install pyspark==3.2.3 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com
3.测试
cd /home/xxx/kdh/spark/bin
./spark-submit --master local[*] /home/xxx/kdh/spark/examples/src/main/python/pi.py 30
四、常见报错
- RuntimeError: Python in worker has different version 3.8 than that in driver 3.6
原因:local模式下,此时使用的是当前环境的python。如果代码里指定了pyspark_env,当前环境也应该是pyspark_env。
conda activate pyspark_env
cd /home/xxx/kdh/spark/bin
./spark-submit --master local[*] /home/xxx/workspace/pyspark_learn/02_pyspark_core/main/02_Wordcount_hdfs.py
五、其它
- arrow安装
pip install pyspark[sql] -i https://pypi.tuna.tsinghua.edu.cn/simple
版权归原作者 kangwq2017 所有, 如有侵权,请联系我们删除。