FLINK on YARN模式
解压安装包:
tar -zvxf flink-1.12.2-bin-scala_2.11.tgz /opt/
修改yarn配置,设置application master重启时尝试的最大次数(cdh内有此参数,默认是2):
<property>
<name>yarn.resourcemanager.am.max-attempts</name>
<value>10</value>
<description>
The maximum number of application master execution attempts.
</description>
配置flink-conf.yaml文件(有些配置已存在,只需要修改参数)
cd /opt/flink-1.12.2/conf/ vi flink-conf.yaml
taskmanager.memory.process.size: 8192m
taskmanager.memory.managed.fraction: 0.2
jobmanager.memory.process.size: 2048m
taskmanager.numberOfTaskSlots: 3
cluster.evenly-spread-out-slots: true
parallelism.default: 10
high-availability: zookeeper
high-availability.storageDir: hdfs:///flink/ha
high-availability.zookeeper.quorum: node1:2181,node2:2181,node3:2181
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 5
rest.port: 8085
web.submit.enable: true
io.tmp.dirs: /tmp
classloader.resolve-order: parent-first
classloader.check-leaked-classloader: false
state.backend: rocksdb
execution.checkpointing.interval: 30000
state.checkpoints.dir: hdfs:///flink/checkpoints
state.checkpoints.num-retained: 10
state.savepoints.dir: hdfs:///flink/savepoints
state.backend.incremental: true
historyserver.web.port: 8082
historyserver.archive.fs.refresh-interval: 10000
jobmanager.execution.failover-strategy: region
修改zoo.cfg:
vi zoo.cfg
#添加zookeeper集群
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
配置环境变量:
vi /etc/profile
export FLINK_HOME=/opt/flink-1.12.2
export PATH=$FLINK_HOME/bin:$PATH
export HADOOP_CLASSPATH=`hadoop classpath`
source /etc/profile
测试
FLINK on YARN示例: yarn-session模式:
#启动成功后,在yarn集群查看,名字是test
bin/yarn-session.sh -nm test -d
#启动flinksql客户端
bin/sql-client.sh embedded -s test
|