[大数据]flink on yarn部署

官网下载对应的版本

flink版本-1.13.6

hadoop版本-3.2.1

步骤：

首先安装jdk

1、配置hosts

vi /etc/hosts

例如：? ? ?

192.168.0.1 flink1
192.168.0.2?flink2
192.168.0.3?flink3

2、防火墙关闭

firewall-cmd --state # 查看是否running
systemctl stop firewalld #停止firewall
systemctl disable firewalld #禁止firewall开机启动

3、ssh免密

? ? ?datanode节点对namenode节点做免密

? ? 命令：

ssh-keygen -t rsa

? ? 生成id_rsa.pub文件?

cat /root/.ssh/id_rsa.pub ?>> /root/.ssh/authorized_keys

? ?各个节点的id_rsa.pub文件添加到authorized_keys

4、配置环境变量

? ? ?vi? /etc/profile

HADOOP_HOME=/opt/hadoop/hadoop-3.2.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_PID_DIR=${HADOOP_HOME}/pids
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.2.1/etc/hadoop

5、hadoop配置文件按功能对应修改

/etc/hadoop
core-site.xml
hadoop-env.sh   
hdfs-site.xml
mapred-env.sh
mapred-site.xml
yarn-env.sh      
yarn-site.xml
workers

格式化namenode

hdfs namenode -format

启动hadoop集群

start-all.sh 
包含start-dfs.sh、start-yarn.sh

?查看标签页返回是否成功或者直接浏览器调用

curl flink1:8088/cluster

flink修改/bin/flink-conf.yaml配置文件对应参数

scp发送到对应的namenode节点

配置flink环境变量 别忘了source一下

FLINK_HOME=/opt/flink/flink-1.13.6
export PATH=$PATH:$FLINK_HOME/bin

命令运行yarn-session.sh -jm 1024 -tm 4096 -s 4 -nm ExamDIMJob -d -st

发现报错如下：

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

1、添加HADOOP_CLASSPATH环境变量

2、由于缺少flink hadoop下载jar包：flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar

添加到flink/lib目录下

再运行又报错：

2022-04-21 14:00:02,794 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Error while running the Flink session.
java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:230) ~[flink-dist_2.11-1.13.6.jar:1.13.6]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:156) ~[flink-dist_2.11-1.13.6.jar:1.13.6]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851) [flink-dist_2.11-1.13.6.jar:1.13.6]

------------------------------------------------------------
 The program finished with the following exception:

java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:230)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:156)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851)

下载jar包：commons-cli-1.5.0.jar?添加到flink/lib目录下

再运行出现如下日志表示已成功

2022-04-21 14:07:16,009 INFO  org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - Default schema
2022-04-21 14:07:16,011 ERROR org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState [] - Authentication failed
2022-04-21 14:07:16,016 INFO  org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager [] - State change: CONNECTED
2022-04-21 14:07:16,184 INFO  org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with ZookeeperLeaderRetrievalDriver{retrievalPath='/leader/rest_server_lock'}.
JobManager Web Interface: http://VM-2-237-centos:8081
2022-04-21 14:07:16,201 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - The Flink YARN session cluster has been started in detached mode. In order to stop Flink gracefully, use the following command:
$ echo "stop" | ./bin/yarn-session.sh -id application_1650509744418_0001
If this should not be possible, then you can also kill Flink via YARN's web interface or via:
$ yarn application -kill application_1650509744418_0001
Note that killing Flink might not clean up all job artifacts and temporary files.

登入hadoop的dashboard页面查看

?点击跳转flink页面

?完工。

阅读世界，共赴山海

423全民读书节，邀你共读