[大数据]Hive（Parquet）存储方式Task个数影响因素

01

写在前面的话

从源码角度探究（Parquet）存储方式，查询语句（不含Shuffle）Task个数影响因素，换句话说也就是Map的分区数。篇幅涉及大量源代码建议收藏后再看。

02

查看hive建表语句

CREATE TABLE `ods.log_info`( 
       log_id   string,
     log_type  string,
     original_msg string,
     create_time bigint,
     update_time bigint)
PARTITIONED BY (                         
  `d_date` string COMMENT '快照日',      
  `pid` string COMMENT '渠道号')       
ROW FORMAT SERDE                         
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'  
STORED AS INPUTFORMAT                              
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'  
OUTPUTFORMAT                                       
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' 

从建表语句中找到INPUTFORMAT?

org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat

03

查看MapredParquetInputFormat类

protected MapredParquetInputFormat(final ParquetInputFormat<ArrayWritable> inputFormat) {
  this.realInput = inputFormat;
  vectorizedSelf = new VectorizedParquetInputFormat(inputFormat);
}

入参ParquetInputFormat<ArrayWritable> inputFormat 为构造方法入参，入口是这个类

04

查看ParquetInputFormat 类（getSplits 方法）

???????

public static boolean isTaskSideMetaData(Configuration configuration) {
        return configuration.getBoolean("parquet.task.side.metadata", Boolean.TRUE);
    }

public List<InputSplit> getSplits(JobContext jobContext) throws IOException {
    Configuration configuration = ContextUtil.getConfiguration(jobContext);
    List<InputSplit> splits = new ArrayList();
    if (!isTaskSideMetaData(configuration)) { // configuration.getBoolean("parquet.task.side.metadata", Boolean.TRUE); 默认返回为True
        splits.addAll(this.getSplits(configuration, this.getFooters(jobContext)));
        return splits;
    } else {
        Iterator i$ = super.getSplits(jobContext).iterator(); // 调用父类的 getSplits 返回的是List<InputSplit> 这里取的是List<InputSplit> 的迭代器


        while(i$.hasNext()) {
            InputSplit split = (InputSplit)i$.next();
            Preconditions.checkArgument(split instanceof FileSplit, "Cannot wrap non-FileSplit: " + split);
            splits.add(ParquetInputSplit.from((FileSplit)split));
        }


        return splits;
    }
}

默认走下面的else分支，即调用super.getSplits方法

public class ParquetInputFormat<T> extends FileInputFormat<Void, T>

05

进入父类FileInputFormat查看（getSplits）

• 获取切片最小?minSize

long?minSize?=?Math.max(getFormatMinSplitSize(),?getMinSplitSize(job));

• ???????

protected long getFormatMinSplitSize() {
  return 1;
}

public static final String SPLIT_MINSIZE = 
  "mapreduce.input.fileinputformat.split.minsize"; 
public static long getMinSplitSize(JobContext job) {
  return job.getConfiguration().getLong(SPLIT_MINSIZE, 1L); 
}

由上面的公式可计算出long?minSize?=?Math.max(1,?1);?minSize?=1

• 计算切片最大maxSize

??long?maxSize?=?getMaxSplitSize(job);?

  public static final String SPLIT_MAXSIZE = 
    "mapreduce.input.fileinputformat.split.maxsize";
  public static long getMaxSplitSize(JobContext context) {
    return context.getConfiguration().getLong(SPLIT_MAXSIZE, 
                                              Long.MAX_VALUE);
  }

如配置了mapreduce.input.fileinputformat.split.maxsize?

参数值的默认值是HDFS集群的块大小，这里值是256

• 找出该路径下的所有文件?listStatus(job)

这个地方会有一个文件名过滤，过滤掉

• 循环获取该目录下的文件大小

• 判断文件是否可切?isSplitable(job, path)、

protected boolean isSplitable(JobContext context, Path filename) {
  return true; // 默认可切
}

• 计算切片大小

long splitSize = computeSplitSize(blockSize, minSize, maxSize);

computeSplitSize方法

protected long computeSplitSize(long blockSize, long minSize,
                                long maxSize) {
??return?Math.max(minSize,?Math.min(maxSize,?blockSize));?
}

方法入参：blockSize?集群块大小（这里是256M）

minSize?最小默认值为1?

maxSize?最大默认值为：

mapreduce.input.fileinputformat.split.maxsize?参数控制

故变成：Math.max(1, Math.min(256, 256))

返回值为：256

完整的getSplits方法如下：

???????

public List<InputSplit> getSplits(JobContext job) throws IOException {
  StopWatch sw = new StopWatch().start();
??//?计算最小值
  long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job)); 
??// 计算最大值
  long maxSize = getMaxSplitSize(job); 


  // generate splits
  List<InputSplit> splits = new ArrayList<InputSplit>();
??//?获取所有的文件会过滤掉以.和_开头的文件
??List<FileStatus>?files?=?listStatus(job);?
???//循环该目录下的所有文件
  for (FileStatus file: files) {
????//?处理文件信息
    Path path = file.getPath();
    long length = file.getLen();
    if (length != 0) {
      BlockLocation[] blkLocations;
      if (file instanceof LocatedFileStatus) {
        blkLocations = ((LocatedFileStatus) file).getBlockLocations();
      } else {
        FileSystem fs = path.getFileSystem(job.getConfiguration());
        blkLocations = fs.getFileBlockLocations(file, 0, length);
      }
??????//?判断文件是否可切?默认值 返回True
??????if?(isSplitable(job,?path))?{?
        long blockSize = file.getBlockSize();
 ????????//?计算切片大小 Math.max(minSize, Math.min(maxSize, blockSize));
????????long?splitSize?=?computeSplitSize(blockSize,?minSize,?maxSize);?
        long bytesRemaining = length;
??????// private static final double SPLIT_SLOP = 1.1;   // 10% slop
?????????//?当文件大小/?splitSize?>1.1时对改文件进行循环切片????
        while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {    
          int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
          splits.add(makeSplit(path, length-bytesRemaining, splitSize,
                      blkLocations[blkIndex].getHosts(),
                      blkLocations[blkIndex].getCachedHosts()));
          bytesRemaining -= splitSize;
        }
????????//?剩下的都是一块
        if (bytesRemaining != 0) {
          int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
          splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
                     blkLocations[blkIndex].getHosts(),
                     blkLocations[blkIndex].getCachedHosts()));
        }
      } else { // not splitable
        splits.add(makeSplit(path, 0, length, blkLocations[0].getHosts(),
                    blkLocations[0].getCachedHosts()));
      }
????}?else?{?//不可切d额时候直接整个加入
      //Create empty hosts array for zero length files
      splits.add(makeSplit(path, 0, length, new String[0]));
    }
  }
  // Save the number of input files for metrics/loadgen
  job.getConfiguration().setLong(NUM_INPUT_FILES, files.size()); // 设置输入的文件个数 
  //public static final String NUM_INPUT_FILES = "mapreduce.input.fileinputformat.numinputfiles";
 
  sw.stop();
  if (LOG.isDebugEnabled()) {
    LOG.debug("Total # of splits generated by getSplits: " + splits.size()
        + ", TimeTaken: " + sw.now(TimeUnit.MILLISECONDS));
  }
  return splits; // 返回所有的切片信息
}

06

核心代码

• 计算最小值

long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));
//由参数mapreduce.input.fileinputformat.split.minsize控制?

• 计算最大值

long maxSize = getMaxSplitSize(job);
//由参数mapreduce.input.fileinputformat.split.maxsize 控制

• 计算splitSize

Math.max(minSize, Math.min(maxSize, blockSize));

• ?HDFS集群默认块大小?blockSize（这里是256M）

07

结论

• 增大Map阶段的分区数

set mapreduce.input.fileinputformat.split.maxsize=128000000;

• 减少Map计算的分区数

set mapreduce.input.fileinputformat.split.maxsize=512000000

08

测试

• 测试SQL:

set?hive.support.quoted.identifiers=None;
set?spark.app.name="ods.log_info"
insert overwrite table ods.log_info_temp partition(d_date= '2021-02-21',pid)
select `(is_settled)?+.+`
from ods.log_info
where 1 > 0

• 结果如下：

set?mapreduce.input.fileinputformat.split.maxsize=128000000;task个数为65

set mapreduce.input.fileinputformat.split.maxsize=256000000；task个数为65

set mapreduce.input.fileinputformat.split.maxsize=512000000；task个数为34

创作打卡挑战赛

赢取流量/现金/CSDN周边激励大奖