0
0

基于容器云提交spark job任务

容器云提交spark job任务

容器云提交Kind=Job类型的spark任务，首先需要申请具有Job任务提交权限的rbac，然后编写对应的yaml文件，通过spark内置的spark-submit命令，提交用户程序(jar包)到集群执行。

1、创建任务job提交权限rbac

创建rbac账户，并分配资源权限，Pod服务账户创建参考,kubernetes api查询命令(kubectl api-resources)；

  1. cat> ecc-recommend-rbac.yaml <<EOF
  2. ---
  3. apiVersion: v1
  4. kind: Namespace
  5. metadata:
  6.   name: item-dev-recommend
  7.   labels:
  8.     name: item-dev-recommend
  9. ---
  10. #基于namespace创建服务账户spark-cdp
  11. apiVersion: v1
  12. kind: ServiceAccount
  13. metadata:
  14.   name: spark-cdp
  15.   namespace: item-dev-recommend
  17. ---
  18. #创建角色资源权限
  19. apiVersion: rbac.authorization.k8s.io/v1
  20. kind: Role
  21. metadata:
  22.   name: spark-cdp
  23.   namespace: item-dev-recommend
  24. rules:
  25.   - apiGroups:
  26.       - ""
  27.     resources:
  28.       - pods
  29.     verbs:
  30.       - '*'
  31.   - apiGroups:
  32.       - ""
  33.     resources:
  34.       - configmaps
  35.     verbs:
  36.       - '*'
  37.   - apiGroups:
  38.       - ""
  39.     resources:
  40.       - services
  41.       - secrets
  42.     verbs:
  43.       - create
  44.       - get
  45.       - delete
  46.   - apiGroups:
  47.       - extensions
  48.     resources:
  49.       - ingresses
  50.     verbs:
  51.       - create
  52.       - get
  53.       - delete
  54.   - apiGroups:
  55.       - ""
  56.     resources:
  57.       - nodes
  58.     verbs:
  59.       - get
  60.   - apiGroups:
  61.       - ""
  62.     resources:
  63.       - resourcequotas
  64.     verbs:
  65.       - get
  66.       - list
  67.       - watch
  68.   - apiGroups:
  69.       - ""
  70.     resources:
  71.       - events
  72.     verbs:
  73.       - create
  74.       - update
  75.       - patch
  76.   - apiGroups:
  77.       - apiextensions.k8s.io
  78.     resources:
  79.       - customresourcedefinitions
  80.     verbs:
  81.       - create
  82.       - get
  83.       - update
  84.       - delete
  85.   - apiGroups:
  86.       - admissionregistration.k8s.io
  87.     resources:
  88.       - mutatingwebhookconfigurations
  89.       - validatingwebhookconfigurations
  90.     verbs:
  91.       - create
  92.       - get
  93.       - update
  94.       - delete
  95.   - apiGroups:
  96.       - sparkoperator.k8s.io
  97.     resources:
  98.       - sparkapplications
  99.       - scheduledsparkapplications
  100.       - sparkapplications/status
  101.       - scheduledsparkapplications/status
  102.     verbs:
  103.       - '*'
  104.   - apiGroups:
  105.       - scheduling.volcano.sh
  106.     resources:
  107.       - podgroups
  108.       - queues
  109.       - queues/status
  110.     verbs:
  111.       - get
  112.       - list
  113.       - watch
  114.       - create
  115.       - delete
  116.       - update
  117.   - apiGroups:
  118.       - batch
  119.     resources:
  120.       - cronjobs
  121.       - jobs
  122.     verbs:
  123.       - '*'   
  124.  
  125. ---
  126. #服务账户spark-cdp绑定角色
  127. apiVersion: rbac.authorization.k8s.io/v1
  128. kind: RoleBinding
  129. metadata:
  130.   name: spark-cdp
  131.   namespace: item-dev-recommend
  132. roleRef:
  133.   apiGroup: rbac.authorization.k8s.io
  134.   kind: Role
  135.   name: spark-cdp
  136. subjects:
  137.   - kind: ServiceAccount
  138.     name: spark-cdp
  139.     
  140. EOF

2、spark pv,pvc

  • 构建pv 挂载NFS，定义pv访问模式(accessModes)和存储容量(capacity)；
  1. cat>ecc-recommend-pv.yaml <<EOF
  2. apiVersion: v1
  3. kind: PersistentVolume
  4. metadata:
  5.   name: dev-cdp-pv01
  6.   namespace: item-dev-recommend
  7. spec:
  8.   capacity:
  9.     storage: 10Gi
  10.   accessModes:
  11.     #访问三种模式：ReadWriteOnce,ReadOnlyMany,ReadWriteMany
  12.     - ReadWriteOnce
  13.   nfs:
  14.     path: /data/nfs
  15.     server: 192.168.0.135
  16.     
  17. EOF
  • 构建pvc
  1. cat>ecc-recommend-pvc.yaml <<EOF
  2. apiVersion: v1
  3. kind: PersistentVolumeClaim
  4. metadata:
  5.   name: dev-cdp-pvc01
  6.   namespace: item-dev-recommend
  7. spec:
  8.   accessModes:
  9.     #匹配模式
  10.     - ReadWriteOnce
  11.   resources:
  12.     requests:
  13.       storage: 10Gi
  15. EOF

3、spark-submit任务提交

将java/scala程序包开发完成后，通过spark-submit命令提交jar包到集群执行。

  1. cat>ecc-recommend-sparksubmit.yaml <<EOF
  2. ---
  3. apiVersion: batch/v1
  4. kind: Job
  5. metadata:
  6.   name: item-recommend-job
  7.   namespace: item-dev-recommend
  8.   labels:
  9.     k8s-app: item-recommend-job
  10. spec:
  11.   template:
  12.     metadata:
  13.       labels:
  14.         k8s-app: item-recommend-job
  15.     spec:
  16.       containers:
  17.         name: item-recommend-job
  18.         - args:
  19.             - /opt/spark/bin/spark-submit
  20.             - --class
  21.             - com.www.ecc.com.recommend.ItemRecommender
  22.             - --master
  23.             - k8s://https:/$(KUBERNETES_SERVICE_HOST):$(KUBERNETES_SERVICE_PORT)
  24.             - --name
  25.             - item-recommend-job
  26.             - --jars
  27.             - /opt/spark/jars/spark-cassandra-connector_2.11-2.3.4.jar
  28.             - --conf
  29.             - spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  30.             - --conf
  31.             - spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token
  32.             - --conf
  33.             - spark.kubernetes.driver.limit.cores=3
  34.             - --conf
  35.             - spark.kubernetes.executor.limit.cores=8
  36.             - --conf
  37.             - spark.kubernetes.driver.limit.memory=5g
  38.             - --conf
  39.             - spark.kubernetes.executor.limit.memory=32g
  40.             - --conf
  41.             - spark.executor.instances=8
  42.             - --conf
  43.             - spark.sql.crossJoin.enable=true
  44.             - --conf
  45.             - spark.executor.cores=6
  46.             - --conf
  47.             - spark.executor.memory=32g
  48.             - --conf
  49.             - spark.driver.cores=3
  50.             - --conf
  51.             - spark.dirver.memory=5g
  52.             - --conf
  53.             - spark.sql.autoBroadcastJoinThreshold=-1
  54.             - --conf
  55.             - spark.kubernetes.namespace=item-dev-recommend
  56.             - --conf
  57.             - spark.driver.port=45970
  58.             - --conf
  59.             - spark.blockManager.port=45980
  60.             - --conf
  61.             - spark.kubernetes.container.image=acpimagehub.ecc.cn/spark:3.11
  62.             - --conf
  63.             - spark.executor.extraJavaOptions="-Duser.timezone=GMT+08:00"
  64.             - --conf
  65.             - spark.driver.extraJavaOptions="-Duser.timezone=GMT+08:00"
  66.             - --conf
  67.             - spark.default.parallelism=500
  68.             - /odsdata/item-recommender-1.0.0-SNAPSHOT.jar
  69.         - env:
  70.             - name: SPARK_SHUFFLE_PARTITIONS
  71.               value: "100"
  72.             - name: CASSANDR_HOST
  73.               value: "192.168.0.1,192.168.0.2,192.168.0.3"
  74.             - name: CASSANDRA_PORT
  75.               value: "9042"
  76.             - name: AUTH_USERNAME
  77.               value: "user"
  78.             - name: AUTH_PASSWORD
  79.               value: "123456"
  80.           image: acpimagehub.ecc.cn/spark:3.11
  81.           imagePullPolicy: IfNotPresent
  82.           ports:
  83.             - containerPort: 9000
  84.               name: 9000tcp2
  85.               protocol: TCP
  86.           resources:
  87.             limits:
  88.               cpu: "3"
  89.               memory: 2Gi
  90.             requests:
  91.               cpu: "3"
  92.               memory: 2Gi
  93.           volumeMounts:
  94.             - mountPath: /odsdata
  95.               name: item-spark-pvc
  96.       volumes:
  97.         - name: item-spark-pvc
  98.           persistentVolumeClaim:
  99.             claimName: dev-cdp-pvc01
  100.       dnsPolicy: ClusterFirst
  101.       restartPolicy: Never
  102.       hostname: item-recommend-job
  103.       securityContext: {}
  104.       serviceAccountName: spark-cdp
  105. ---
  106. apiVersion: v1
  107. kind: Service
  108. metadata:
  109.   name: item-recommend-job
  110.   namespace: item-dev-recommend
  111. spec:
  112.   type: NodePort
  113.   ports:
  114.     - name: sparkjob-tcp4040
  115.       port: 4040
  116.       protocol: TCP
  117.       targetPort: 4040
  118.     #spark driver port
  119.     - name: sparkjob-tcp-45970
  120.       port: 45970
  121.       protocol: TCP
  122.       targetPort: 45970
  123.     #spark ui
  124.     - name: sparkjob-tcp-48080
  125.       port: 48080
  126.       protocol: TCP
  127.       targetPort: 48080
  128.     #spark executor port
  129.     - name: sparkjob-tcp-45980
  130.       port: 45980
  131.       protocol: TCP
  132.       targetPort: 45980
  133.   selector:
  134.     k8s-app: item-recommend-job
  136. EOF

4、打包插件小记

  1. <build><resources><resource><directory>src/main/resources</directory><includes><include>*.properties</include></includes><filtering>false</filtering></resource></resources><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><configuration><skipTests>true</skipTests></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>3.6.1</version><configuration><source>${java.version}</source><target>${java.version}</target><encoding>${project.build.sourceEncoding}</encoding></configuration><executions><execution><phase>compile</phase><goals><goal>compile</goal></goals></execution></executions></plugin><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.2.2</version><executions><execution><id>scala-compile-first</id><phase>process-resources</phase><goals><goal>add-source</goal><goal>compile</goal><goal>testCompile</goal></goals></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>3.2.1</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals></execution></executions></plugin></plugins></build>
标签： spark scala 大数据
本文转载自: https://blog.csdn.net/software444/article/details/129337814
版权归原作者 茅台技术人 所有， 如有侵权，请联系我们删除。
登录后发布评论

“基于容器云提交spark job任务”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？