自己写着玩的时候遇到的问题在此整理一下,要能捎带手帮各位看官解了惑那真是倍感荣幸,Flink版本1.12.0
IDEA调试Flink任务时需要WebUI
1.问题描述
在idea进行flink任务开发时,希望可以通过web ui查看一些信息
2.原因定位
在本地调试的时候时通过StreamExecutionEnvironment.createLocalEnvironment()创建执行时上下文,该方法未启动 web monitoring UI,需要通过createLocalEnvironmentWithWebUI(Configuration)方法创建上下文,该方法不仅会创建本地执行环境,同时会启动web monitoring UI
3.解决方法
添加依赖
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_${scala.version}</artifactId>
<version>${flink.version}</version>
</dependency>
通过createLocalEnvironmentWithWebUI(Configuration)方法创建执行时上下文
Configuration conf = new Configuration();
conf.setInteger(RestOptions.PORT, 12345);
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
如果不指定端口号可不可以?答案是可以!If the configuration key 'rest.port' was set in the configuration, that particular port will be used for the web UI. Otherwise, the default port (8081) will be used.
IDEA调试FLink任务时想看运行日志
1.问题描述
在idea调试flink任务时候想看日志,但是控制台只有三行干巴巴的提示,这时候该怎么办?不要慌,人家都给你链接提示你了
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2.原因定位
点进链接,我们找到对应的原因,翻译一下就是少包,那咱就加一下
3.解决方法
添加依赖
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
</dependency>
添加完依赖以后,控制台的日志就像便秘以后抹了开塞露,裤衩裤衩都显示出来了。
checkpoint失败
1.问题描述
在本地调试flink任务时,发现状态checkpoint失败
2.原因定位
查看日志发现状态大小超过默认最大值,由于代码中没有指定StateBackend,因此默认使用MemoryStateBackend ,内存中默认为5MB,The default maximal size that the snapshotted memory state may have (5 MiBytes).
java.util.concurrent.ExecutionException: java.io.IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=846910422 , maxSize=5242880 . Consider using a different state backend, like the File System State backend.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:583)
at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:53)
at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:115)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=846910422 , maxSize=5242880 . Consider using a different state backend, like the File System State backend.
at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory.checkSize(MemCheckpointStreamFactory.java:64)
at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetBytes(MemCheckpointStreamFactory.java:145)
at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetHandle(MemCheckpointStreamFactory.java:126)
at org.apache.flink.runtime.state.CheckpointStreamWithResultProvider$PrimaryStreamOnly.closeAndFinalizeCheckpointStreamResult(CheckpointStreamWithResultProvider.java:77)
at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:199)
at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:158)
at org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:75)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:580)
... 5 more
3.解决方法
MemoryStateBackend 类中可以指定maxStateSize 大小,可通过MemoryStateBackend(int maxStateSize) 方法进行指定,方法如下
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
env.setStateBackend(new MemoryStateBackend(1048576000));
在进行大状态存储时还是推荐使用其他的StateBackend方式,不建议使用MemoryStateBackend,具体原因可参考官网,不在此处赘述。
|