今天在做MapReduce的时候,为了做个降序排列,所以定义了一个自定义序列,然后运行的时候就出现了这个问题,查了好多教程,最后发现错在了一个比较离谱的地方,记录一下,顺带写上其他几种可能。(1-3来源网络,4亲身经历) 先放个报错信息以供参考
2022-04-29 01:04:34,361 WARN [LocalJobRunner Map Task Executor
java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
... 16 more
2022-04-29 01:04:34,364 INFO [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2022-04-29 01:04:34,370 WARN [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1220408697_0001
java.lang.Exception: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:424)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
... 10 more
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
... 16 more
情况1
包导入错了,这是网上最多情况的报错原因,比如Text 应该要导入org.apache.hadoop.io.Text 之类的,仔细检查一下,看看有没有。
情况2
自定义序列的时候没有没有完全继承Writable 和Comparable , 这种情况的话,只要继承一下WritableComparable 就ok了, WritableComparable 已经继承了Writable 和Comparable 。比如我的自定义序列:
public class classes implements WritableComparable<classes>{...}
情况3
如果是自定义序列的话,再MapReduce操作中是要求其可迭代的,也就是说需要重写一个compareTo 方法,比如我的
public int compareTo(classes o) {
return (int) (-this.avgsorce+o.getAvgsorce());
}
如果没写的话也可能出现报错
情况4
最后就是我的问题了,很简单,在自定义序列的时候,我只定义了一个有参的构造器,总所周知,在Java里如果已经定义了构造器,系统就不再有默认构造器了,这就导致我定义的这个自定义序列没有无参构造器,好巧不巧,MapReduce排序的时候就偏偏要用到这个无参构造器,得嘞,这可不就报错了,到此我定义了一个无参构造器,问题解决
public classes(){
}
这坑有点深,找了我好久,再加上现在是北京时间1:21,半夜啦!!忍不住发一下吐槽一下(这句纯吐槽,不想看可以略过哈哈哈)
转载请注明出处,谢谢!
|