|
我的原创地址：https://dongkelun.com/2020/11/17/hadoopExcepetion2/
前言
这个异常发生在Spark读取Windows本地CSV然后show，当然一般情况下不会发生，还有一个条件，项目里加了hbase-client和hbase-mapreduce，具体是哪一个依赖或者两个依赖合起来造成的影响我没有去细究，主要记录解决方法
网上也有其他很多情况可能出现这个异常
详细异常信息
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:640)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1223)
at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1428)
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles(InMemoryFileIndex.scala:281)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:177)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:176)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:176)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:127)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:91)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init>(InMemoryFileIndex.scala:67)
at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$createInMemoryFileIndex(DataSource.scala:533)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:371)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:619)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:468)
at com.inspur.demo.spark.JavaCSV2ES.main(JavaCSV2ES.java:46)
解决方法
先讲最终有效的解决方法：
在https://github.com/steveloughran/winutils下载对应版本的hadoop.dll文件放到C:\Windows\System32下即可解决
更新：2021-04-27
因上面github地址对应的hadoop版本不是很全，若想下载hadoop3.1或3.2版本等其他版本，可以从以下地址下载
https://github.com/dongkelun/winutils
网上其他方法 ：
1、本地安装hadoop并将hadoop.dll 重启电脑 https://my.oschina.net/u/4307631/blog/4012671
亲测无效，不知道其他情况是否有效，反正我这种情况无效
2、改源码 https://www.cnblogs.com/kelly-one/p/10514371.html
嫌麻烦没有测试，因为这种方法每个项目都要新建一个类，太麻烦了，所以不到最后不想尝试
放到C:\Windows\System32的思路参考：http://www.bubuko.com/infodetail-1092966.html
|