0
0

CDH 6.3.2集成flink 1.18 zookeeper版本不匹配Flink-yarn启动失败

CDH 6.3.2集成flink 1.18 zookeeper版本不匹配Flink-yarn不能正常启动，而在CHD Web页面，flink日志报错提示不明确，不能定位具体错误。CM WEB启动失败错误日志如下图所示：
CDH
CM查看完成错误日志

  1. [31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
  2. [31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  3. [31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 260009
  4. 2023年 12月 31日 星期日 10:45:09 CST
  5. + locate_java_home
  6. + locate_java_home_no_verify
  7. + JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
  8. + local JAVA11_HOME_CANDIDATES
  9. + OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
  10. + local OPENJAVA11_HOME_CANDIDATES
  11. + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
  12. + local JAVA8_HOME_CANDIDATES
  13. + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
  14. + local OPENJAVA8_HOME_CANDIDATES
  15. + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
  16. + local MISCJAVA_HOME_CANDIDATES
  17. + case ${BIGTOP_JAVA_MAJOR} in
  18. + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
  19. + '[' -z '' ']'
  20. + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
  21. ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
  22. + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
  23. + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
  24. + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  25. + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  26. + break 2
  27. + verify_java_home
  28. + '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
  29. + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  30. + '[' -n '' ']'
  31. + source_parcel_environment
  32. + '[' '!' -z '' ']'
  33. + echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
  34. + echo 'Using scripts/control.sh as process script'
  35. + replace_conf_dir
  36. + echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  37. + echo CMF_CONF_DIR=
  38. + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
  39. ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
  40. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
  41. + make_scripts_executable
  42. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
  43. + RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  44. + '[' '' == true ']'
  45. + chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
  46. + export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  47. + COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  48. + exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
  49. + USAGE='Usage: control.sh (start|stop)'
  50. + OPERATION=start
  51. + case $OPERATION in
  52. ++ hostname -f
  53. + NODE_HOST=node01
  54. + '[' '!' -d /opt/cloudera/parcels/FLINK ']'
  55. + FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
  56. + FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
  57. + '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
  58. + mkdir /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
  59. + cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
  60. + sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  61. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  62. ++ grep high-availability:
  63. + HIGH_MODE=
  64. + '[' '' = '' ']'
  65. + echo 'high-availability: zookeeper'
  66. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  67. ++ grep high-availability:
  68. + HIGH_MODE='high-availability: zookeeper'
  69. + echo 'HIGH_MODE: high-availability: zookeeper'
  70. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  71. ++ grep high-availability.zookeeper.quorum:
  72. + HIGH_ZK_QUORUM=
  73. + '[' '' = '' ']'
  74. + echo 'high-availability.zookeeper.quorum: node01:2181'
  75. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  76. ++ grep high-availability.zookeeper.quorum:
  77. + HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
  78. + echo 'HIGH_ZK_QUORUM: high-availability.zookeeper.quorum: node01:2181'
  79. + cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
  80. + HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  81. + export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
  82. + echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  83. + echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  84. + echo ''
  85. ++ date
  86. + echo 'Date: 2023年 12月 31日 星期日 10:45:09 CST'
  87. + echo 'Host: node01'
  88. + echo 'NODE_TYPE: '
  89. + echo 'ZK_QUORUM: node01:2181'
  90. + echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
  91. + echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
  92. + echo ''
  93. + '[' true = true ']'
  94. + exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
  95. [31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
  96. [31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  97. [31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 262546
  98. 2023年 12月 31日 星期日 10:45:27 CST
  99. + locate_java_home
  100. + locate_java_home_no_verify
  101. + JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
  102. + local JAVA11_HOME_CANDIDATES
  103. + OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
  104. + local OPENJAVA11_HOME_CANDIDATES
  105. + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
  106. + local JAVA8_HOME_CANDIDATES
  107. + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
  108. + local OPENJAVA8_HOME_CANDIDATES
  109. + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
  110. + local MISCJAVA_HOME_CANDIDATES
  111. + case ${BIGTOP_JAVA_MAJOR} in
  112. + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
  113. + '[' -z '' ']'
  114. + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
  115. ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
  116. + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
  117. + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
  118. + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  119. + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  120. + break 2
  121. + verify_java_home
  122. + '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
  123. + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  124. + '[' -n '' ']'
  125. + source_parcel_environment
  126. + '[' '!' -z '' ']'
  127. + echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
  128. + echo 'Using scripts/control.sh as process script'
  129. + replace_conf_dir
  130. + echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  131. + echo CMF_CONF_DIR=
  132. + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
  133. ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
  134. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
  135. + make_scripts_executable
  136. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
  137. + RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  138. + '[' '' == true ']'
  139. + chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
  140. + export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  141. + COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  142. + exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
  143. + USAGE='Usage: control.sh (start|stop)'
  144. + OPERATION=start
  145. + case $OPERATION in
  146. ++ hostname -f
  147. + NODE_HOST=node01
  148. + '[' '!' -d /opt/cloudera/parcels/FLINK ']'
  149. + FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
  150. + FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
  151. + '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
  152. + rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
  153. + cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
  154. + sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  155. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  156. ++ grep high-availability:
  157. + HIGH_MODE='high-availability: zookeeper'
  158. + '[' 'high-availability: zookeeper' = '' ']'
  159. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  160. ++ grep high-availability.zookeeper.quorum:
  161. + HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
  162. + '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
  163. + cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
  164. + HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  165. + export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
  166. + echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  167. + echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  168. + echo ''
  169. ++ date
  170. + echo 'Date: 2023年 12月 31日 星期日 10:45:27 CST'
  171. + echo 'Host: node01'
  172. + echo 'NODE_TYPE: '
  173. + echo 'ZK_QUORUM: node01:2181'
  174. + echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
  175. + echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
  176. + echo ''
  177. + '[' true = true ']'
  178. + exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
  179. [31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
  180. [31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  181. [31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 264012
  182. 2023年 12月 31日 星期日 10:45:42 CST
  183. + locate_java_home
  184. + locate_java_home_no_verify
  185. + JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
  186. + local JAVA11_HOME_CANDIDATES
  187. + OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
  188. + local OPENJAVA11_HOME_CANDIDATES
  189. + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
  190. + local JAVA8_HOME_CANDIDATES
  191. + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
  192. + local OPENJAVA8_HOME_CANDIDATES
  193. + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
  194. + local MISCJAVA_HOME_CANDIDATES
  195. + case ${BIGTOP_JAVA_MAJOR} in
  196. + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
  197. + '[' -z '' ']'
  198. + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
  199. ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
  200. + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
  201. + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
  202. + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  203. + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  204. + break 2
  205. + verify_java_home
  206. + '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
  207. + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  208. + '[' -n '' ']'
  209. + source_parcel_environment
  210. + '[' '!' -z '' ']'
  211. + echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
  212. + echo 'Using scripts/control.sh as process script'
  213. + replace_conf_dir
  214. + echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  215. + echo CMF_CONF_DIR=
  216. + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
  217. ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
  218. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
  219. + make_scripts_executable
  220. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
  221. + RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  222. + '[' '' == true ']'
  223. + chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
  224. + export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  225. + COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  226. + exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
  227. + USAGE='Usage: control.sh (start|stop)'
  228. + OPERATION=start
  229. + case $OPERATION in
  230. ++ hostname -f
  231. + NODE_HOST=node01
  232. + '[' '!' -d /opt/cloudera/parcels/FLINK ']'
  233. + FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
  234. + FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
  235. + '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
  236. + rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
  237. + cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
  238. + sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  239. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  240. ++ grep high-availability:
  241. + HIGH_MODE='high-availability: zookeeper'
  242. + '[' 'high-availability: zookeeper' = '' ']'
  243. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  244. ++ grep high-availability.zookeeper.quorum:
  245. + HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
  246. + '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
  247. + cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
  248. + HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  249. + export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
  250. + echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  251. + echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  252. + echo ''
  253. ++ date
  254. + echo 'Date: 2023年 12月 31日 星期日 10:45:42 CST'
  255. + echo 'Host: node01'
  256. + echo 'NODE_TYPE: '
  257. + echo 'ZK_QUORUM: node01:2181'
  258. + echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
  259. + echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
  260. + echo ''
  261. + '[' true = true ']'
  262. + exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
  263. [31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
  264. [31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  265. [31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 265980
  266. 2023年 12月 31日 星期日 10:45:59 CST
  267. + locate_java_home
  268. + locate_java_home_no_verify
  269. + JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
  270. + local JAVA11_HOME_CANDIDATES
  271. + OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
  272. + local OPENJAVA11_HOME_CANDIDATES
  273. + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
  274. + local JAVA8_HOME_CANDIDATES
  275. + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
  276. + local OPENJAVA8_HOME_CANDIDATES
  277. + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
  278. + local MISCJAVA_HOME_CANDIDATES
  279. + case ${BIGTOP_JAVA_MAJOR} in
  280. + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
  281. + '[' -z '' ']'
  282. + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
  283. ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
  284. + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
  285. + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
  286. + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  287. + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  288. + break 2
  289. + verify_java_home
  290. + '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
  291. + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
  292. + '[' -n '' ']'
  293. + source_parcel_environment
  294. + '[' '!' -z '' ']'
  295. + echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
  296. + echo 'Using scripts/control.sh as process script'
  297. + replace_conf_dir
  298. + echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  299. + echo CMF_CONF_DIR=
  300. + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
  301. ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
  302. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
  303. + make_scripts_executable
  304. + find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
  305. + RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  306. + '[' '' == true ']'
  307. + chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
  308. + export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  309. + COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
  310. + exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
  311. + USAGE='Usage: control.sh (start|stop)'
  312. + OPERATION=start
  313. + case $OPERATION in
  314. ++ hostname -f
  315. + NODE_HOST=node01
  316. + '[' '!' -d /opt/cloudera/parcels/FLINK ']'
  317. + FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
  318. + FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
  319. + '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
  320. + rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
  321. + cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
  322. + sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  323. ++ grep high-availability:
  324. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  325. + HIGH_MODE='high-availability: zookeeper'
  326. + '[' 'high-availability: zookeeper' = '' ']'
  327. ++ grep high-availability.zookeeper.quorum:
  328. ++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
  329. + HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
  330. + '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
  331. + cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
  332. + HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  333. + export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
  334. + echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
  335. + echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
  336. + echo ''
  337. ++ date
  338. + echo 'Date: 2023年 12月 31日 星期日 10:45:59 CST'
  339. + echo 'Host: node01'
  340. + echo 'NODE_TYPE: '
  341. + echo 'ZK_QUORUM: node01:2181'
  342. + echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
  343. + echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
  344. + echo ''
  345. + '[' true = true ']'
  346. + exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming

CM管理页面查看异常日志，始终不能定位启动异常原因，此时查看flink详情启动日志，日志文件/var/log/flink/flink-yarn.out，详细错误日志如下图所示：
在这里插入图片描述
完整错误日志信息：

  1. SLF4J: Class path contains multiple SLF4J bindings.
  2. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/FLINK-1.18.0-BIN-SCALA_2.12/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  3. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  4. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/avro-tools-1.8.2-cdh6.3.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  8. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/parquet-tools-1.9.0-cdh6.3.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  9. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  10. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  11. ERROR StatusLogger Reconfiguration failed: No configuration found for '7ea987ac' at 'null' in 'null'
  12. 22:57:15.334 [main] ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while running the Flink session.
  13. org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
  14.         at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:444) ~[flink-dist-1.18.0.jar:1.18.0]
  15.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:604) ~[flink-dist-1.18.0.jar:1.18.0]
  16.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:865) ~[flink-dist-1.18.0.jar:1.18.0]
  17.         at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
  18.         at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
  19.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) ~[avro-tools-1.8.2-cdh6.3.2.jar:1.8.2-cdh6.3.2]
  20.         at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.18.0.jar:1.18.0]
  21.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:865) [flink-dist-1.18.0.jar:1.18.0]
  22. Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
  23. Diagnostics from YARN: Application application_1703945118355_0021 failed 1 times in previous 10000 milliseconds (global limit =2; local limit is =1) due to AM Container for appattempt_1703945118355_0021_000001 exited with  exitCode: 239
  24. Failing this attempt.Diagnostics: [2023-12-30 22:57:14.939]Exception from container-launch.
  25. Container id: container_1703945118355_0021_01_000001
  26. Exit code: 239
  28. [2023-12-30 22:57:14.942]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
  29. Last 4096 bytes of prelaunch.err :
  31. [2023-12-30 22:57:14.943]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
  32. Last 4096 bytes of prelaunch.err :
  34. For more detailed output, check the application tracking page: http://node01:8088/cluster/app/application_1703945118355_0021 Then click on links to logs of each attempt.
  35. . Failing the application.
  36. If log aggregation is enabled on your cluster, use this command to further investigate the issue:
  37. yarn logs -applicationId application_1703945118355_0021
  38.         at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1262) ~[flink-dist-1.18.0.jar:1.18.0]
  39.         at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:633) ~[flink-dist-1.18.0.jar:1.18.0]
  40.         at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437) ~[flink-dist-1.18.0.jar:1.18.0]
  41.         ... 7 more
  43. ------------------------------------------------------------
  44.  The program finished with the following exception:
  46. org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
  47.         at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:444)
  48.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:604)
  49.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:865)
  50.         at java.security.AccessController.doPrivileged(Native Method)
  51.         at javax.security.auth.Subject.doAs(Subject.java:422)
  52.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
  53.         at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
  54.         at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:865)
  55. Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
  56. Diagnostics from YARN: Application application_1703945118355_0021 failed 1 times in previous 10000 milliseconds (global limit =2; local limit is =1) due to AM Container for appattempt_1703945118355_0021_000001 exited with  exitCode: 239
  57. Failing this attempt.Diagnostics: [2023-12-30 22:57:14.939]Exception from container-launch.
  58. Container id: container_1703945118355_0021_01_000001
  59. Exit code: 239
  61. [2023-12-30 22:57:14.942]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
  62. Last 4096 bytes of prelaunch.err :
  64. [2023-12-30 22:57:14.943]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
  65. Last 4096 bytes of prelaunch.err :
  67. For more detailed output, check the application tracking page: http://node01:8088/cluster/app/application_1703945118355_0021 Then click on links to logs of each attempt.
  68. . Failing the application.
  69. If log aggregation is enabled on your cluster, use this command to further investigate the issue:
  70. yarn logs -applicationId application_1703945118355_0021
  71.         at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1262)
  72.         at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:633)
  73.         at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437)
  74.         ... 7 more

分析日志信息，猜测可能是flink 1.18.0和zookeeper版本不兼容，因为CDH默认zookeeper版本是zookeeper-3.4.6，而flink1.18可能因为版本较高，所以需要更高版本的zookeeper，因此考虑升级CDH的zookeeper版本到zookeeper 3.8。

升级步骤
1.maven仓库下载Cloudeara版本的zookeeper jar包，仓库地址：https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper?repo=cloudera-repos，如下图所示：
在这里插入图片描述
2.下载zookeeper 3.8
下载地址：https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gz

3.上传Cloudeara版本的zookeeper jar到CM下，即把zookeeper-3.8.1.7.1.9.0-387.jar包上传到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/下，并备份旧版本zookeeper即可。
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/
mv zookeeper-3.4.5-cdh6.3.2.jar zookeeper-3.4.5-cdh6.3.2.jar.bak
mv zookeeper-3.4.5.jar zookeeper-3.4.5.jar.bak
如下图所示：
在这里插入图片描述
4.备份旧版本zookeeper的lib，上传下载apache-zookeeper-3.8.3-bin的lib目录下的jar包到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper目录下
如下图所示：
在这里插入图片描述
在这里插入图片描述
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper
备份zookeeper的lib，mv lib lib.bak
上传apache-zookeeper-3.8.3-bin下的lib
如下图所示：
在这里插入图片描述
重新启动Flink-yarn，启动成功。
在这里插入图片描述
5.相关大数据学习demo地址:
https://github.com/carteryh/big-data

标签： flink zookeeper 大数据
本文转载自: https://blog.csdn.net/qq_21480329/article/details/135313722
版权归原作者 qq_21480329 所有， 如有侵权，请联系我们删除。
登录后发布评论

“CDH 6.3.2集成flink 1.18 zookeeper版本不匹配Flink-yarn启动失败”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？

文章导航
没有找到文章内容分段。。。