[大数据]spark conf 配置优化

一、背景

spark版本：2.3.1
scala版本：2.11.8

二、conf配置说明

三、conf设置方式

3.1、代码配置

scala 两种设置如下

import org.apache.spark.sql.SparkSession


val spark: SparkSession = SparkSession.builder()
  .config(
    "hive.metastore.client.factory.class",
    "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
  ) // aws Glue 数据单元管理
  .enableHiveSupport()
  .config("hive.exec.dynamici.partition", true)
  .config("hive.exec.dynamic.partition.mode", "nonstrict")
  .getOrCreate()
  
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

3.2、提交形式

spark-submit \
--name conf_example \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--executor-cores 1 \
--executor-memory 1G \
--driver-memory 1G \
--class xxx.xxxx.xxxxx.xxx.xxxx \
--files conf.properties,log4j.properties,log4j2.xml \
--conf spark.hadoop.ipc.client.fallback-to-simple-auth-allowed=true \
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties \
--jars sss.jar,wwqq.jar \
main.jar