前言

本文隶属于专栏《Spark异常问题汇总》，该专栏为笔者原创，引用请注明来源，不足和错误之处请在评论区帮忙指出，谢谢！

本专栏目录结构和参考文献请见 Spark异常问题汇总

正文

执行 spark-sql 脚本的时候报错：

[hadoop@node2 ~]$ spark-sql
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/07/31 22:19:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
        at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:724)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:654)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:548)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:952)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

查看权限

[hadoop@node2 ~]$ hadoop fs -ls -R /tmp
drwxr-xr-x   - hadoop supergroup          0 2021-06-20 11:16 /tmp/hive
# 其他省略

尝试解决

[hadoop@node2 ~]$ hadoop fs -chmod 777 /tmp/hive
[hadoop@node2 ~]$ hadoop fs -ls -R /tmp
drwxrwxrwx   - hadoop supergroup          0 2021-06-20 11:16 /tmp/hive
# 其他省略

可以看到目录权限已经变更了，再次执行

[hadoop@node2 ~]$ spark-sql
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/07/31 22:29:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
        at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:724)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:654)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:548)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:952)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

说实话，刚开始我有点慌了，明明权限都已经修改了，但是还是报同样的错误，于是我去另外两台机器上面查看一下权限，结果都是正常的

[hadoop@node1 ~]$ hadoop fs -ls -R /tmp
drwxrwxrwx   - hadoop supergroup          0 2021-06-20 11:16 /tmp/hive
# 其他省略

[hadoop@node3 ~]$ hadoop fs -ls -R /tmp
drwxrwxrwx   - hadoop supergroup          0 2021-06-20 11:16 /tmp/hive
# 其他省略

最终解决

[hadoop@node3 ~]$ spark-sql
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/07/31 22:32:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/07/31 22:32:06 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/07/31 22:32:06 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/07/31 22:32:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/07/31 22:32:09 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore hadoop@172.16.68.203
21/07/31 22:32:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
Spark master: local[*], Application Id: local-1627741924821
spark-sql>