IT数码 购物 网址 头条 软件 日历 阅读 图书馆
TxT小说阅读器
↓语音阅读,小说下载,古典文学↓
图片批量下载器
↓批量下载图片,美女图库↓
图片自动播放器
↓图片自动播放器↓
一键清除垃圾
↓轻轻一点,清除系统垃圾↓
开发: C++知识库 Java知识库 JavaScript Python PHP知识库 人工智能 区块链 大数据 移动开发 嵌入式 开发工具 数据结构与算法 开发测试 游戏开发 网络协议 系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑 笔记本 显卡 显示器 固态硬盘 硬盘 耳机 手机 iphone vivo oppo 小米 华为 单反 装机 图拉丁
 
   -> 大数据 -> Flink On Yarn Pre-job提交空指针报错 -> 正文阅读

[大数据]Flink On Yarn Pre-job提交空指针报错

Flink On Yarn Pre-job提交空指针报错

背景

通过命令提交job到yarn是正常运行的,命令如下:

./bin/flink run -d -m yarn-cluster ./examples/streaming/TopSpeedWindowing.jar

项目需求通过代码的形式提交,这边方便获取jobId和applicationId,如是找了一个开源的代码,代码地址flink-spark-submiter,倒腾了半天终于可以提交了,执行如下:

java -jar flink-yarn-submiter-1.0-SNAPSHOT.jar -hadoopConfDir /root/app/hadoop/etc/hadoop -name test-one -flinkConfDir /opt/topology/service/flink/conf -flinkJarPath /opt/topology/service/flink/lib  -runJarPath /opt/topology/service/flink/examples/streaming/TopSpeedWindowing.jar

yarn也收到提交申请,如图:
在这里插入图片描述
yarn这块状态从一开始的ACCEPTED变为RUNNING,但是过会儿就失败了,查看日志Flink集群启动失败,具体错误如下:

2022-02-25 09:20:03,186 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Fatal error occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job 1071fbecc2f79b948678f19dc58652fa failed.
	at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:910) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:473) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:450) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$3(Dispatcher.java:427) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_281]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) [?:1.8.0_281]
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: java.util.concurrent.CompletionException: java.lang.NullPointerException
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: java.lang.NullPointerException
	at org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory$PartitionLocationConstraint.fromJobType(TaskDeploymentDescriptorFactory.java:328) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:339) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:190) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:130) ~[flink-dist_2.11-1.14.2.jar:1.14.2]

问题排查

从截图上面看,是由于空指针导致jobMaster启动失败了,具体位置TaskDeploymentDescriptorFactory.java:328,查看源码如下:
在这里插入图片描述
查看类文件注意版本,版本不同类可能不一样。
再往上找看下JobType是哪里来的,如图:
在这里插入图片描述
可以看到是JobGraph对象下的属性,查看flink-spark-submiter里面的代码,看下是不是没有赋值,发现flink-spark-submiter里面的JobGraph对象是没有jobType这个属性的,再对比发现flink-spark-submiter用的flink版本是1.12.1,而我们部署的版本是1.14.2,升级flink-spark-submiter的flink版本后该问题得到解决。

  大数据 最新文章
实现Kafka至少消费一次
亚马逊云科技:还在苦于ETL?Zero ETL的时代
初探MapReduce
【SpringBoot框架篇】32.基于注解+redis实现
Elasticsearch:如何减少 Elasticsearch 集
Go redis操作
Redis面试题
专题五 Redis高并发场景
基于GBase8s和Calcite的多数据源查询
Redis——底层数据结构原理
上一篇文章      下一篇文章      查看所有文章
加:2022-02-26 11:37:32  更:2022-02-26 11:38:50 
 
开发: C++知识库 Java知识库 JavaScript Python PHP知识库 人工智能 区块链 大数据 移动开发 嵌入式 开发工具 数据结构与算法 开发测试 游戏开发 网络协议 系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑 笔记本 显卡 显示器 固态硬盘 硬盘 耳机 手机 iphone vivo oppo 小米 华为 单反 装机 图拉丁

360图书馆 购物 三丰科技 阅读网 日历 万年历 2024年11日历 -2024/11/24 11:58:38-

图片自动播放器
↓图片自动播放器↓
TxT小说阅读器
↓语音阅读,小说下载,古典文学↓
一键清除垃圾
↓轻轻一点,清除系统垃圾↓
图片批量下载器
↓批量下载图片,美女图库↓
  网站联系: qq:121756557 email:121756557@qq.com  IT数码