问题
在spark集群运行scala程序报错
Error:scalac: error while loading package, class file 'C:\Program Files (x86)\scala\lib\scala-library.jar(scala/collection/immutable/package.class)' is broken
(class java.lang.RuntimeException/error reading Scala signature of package.class: Scala signature package has wrong version
**expected: 5.0**
found: 5.2 in package.class)
程序内容
object ActionDemo {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("ActionDemo").setMaster("spark://192.168.137.15:7077")
val context = new SparkContext(conf)
val rdd1:RDD[Int] = context.parallelize(List(1,2,3,4,5,6,4))
println(rdd1.collect().toList)
println(rdd1.count())
val rdd2:Array[Int]=rdd1.top(3)
println(rdd2.toBuffer)
val rdd3:Array[Int]=rdd1.take(3)
println(rdd3.toBuffer)
}
}
原因 spark-core版本需要和scala版本一致
因为scala版本是这样的
<scala.version>2.12.15</scala.version>
所以需要修改spark-core依赖，其他的依赖也需要修改为_2.12版本
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.2.1</version>
</dependency>
完整maven
<!-- 声明公有的属性 -->
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.12.15</scala.version>
<spark.version>3.2.1</spark.version>
<hadoop.version>2.7.1</hadoop.version>
<mysql.version>8.0.13</mysql.version>
</properties>
<!-- 声明并引入公有的依赖 -->
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive-thriftserver_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
