前言

MergeTree Family启动慢问题一直是ClickHouse社区的痛点。在完成对所有MergeTree part的加载之前，clickhouse-server实例无法对外提供服务。而在生产环境中，加载过程动辄持续几十分钟，严重影响clickhouse集群的可用性。

针对这个痛点，我们开发了MergeTree启动加速特性：通过RocksDB缓存part元数据，将clickhouse-server的启动时间从小时级减少到分钟级。该特性目前已被合入社区，PR见：https://github.com/ClickHouse/ClickHouse/pull/32928。本文将着重讲解如何使用该特性，并对启动性能进行评测。

使用

配置

首先需要在config.xml中加入配置

?<!--?Uncomment?if?enable?merge?tree?metadata?cache?-->
????<merge_tree_metadata_cache>
????????<lru_cache_size>268435456</lru_cache_size>
????????<continue_if_corrupted>true</continue_if_corrupted>
????</merge_tree_metadata_cache>

lru_cache_size: 用于缓存part元数据的RocksDB实例中的LRU大小

continue_if_corrupted: 当本地的RocksDB目录读取失败时，用户可选择退出进程(false)或者清空脏数据(true)。

建表

创建MergeTree Family的表时，需指定settings: use_metadata_cache = true，这样该表下所有part的元数据除了落地到文件中，还会写一份到RocksDB.

建表

CREATE?TABLE?test
(
????`x`?UInt32,?
????`y`?UInt32,?
????`z`?UInt32,?
????`t`?UInt32
)
ENGINE?=?MergeTree
PARTITION?BY?x?%?10
ORDER?BY?(x,?y)?
SETTINGS?use_metadata_cache?=?true

插入数据

insert?into?test?select?number,?number,?number,?number?from?system.numbers?limit?100;

查看元数据缓存

:)?desc?system.merge_tree_metadata_cache?

DESCRIBE?TABLE?system.merge_tree_metadata_cache

Query?id:?d2a96900-db4f-45ef-8d19-f1d2f6edd930

┌─name──┬─type───┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│?key???│?String?│??????????????│????????????????????│?????????│??????????????????│????????????????│
│?value?│?String?│??????????????│????????????????????│?????????│??????????????????│????????????????│
└───────┴────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘


:)?select?key?from?system.merge_tree_metadata_cache?where?key?like?'default:%'?limit?10;?

SELECT?key
FROM?system.merge_tree_metadata_cache
WHERE?key?LIKE?'default:%'
LIMIT?10

Query?id:?7c87cb78-ba72-4702-b457-6bdfba5ee83c

┌─key──────────────────────────────────────────────────────────────────────────────────────────┐
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/checksums.txt?????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/columns.txt???????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/count.txt?????????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/default_compression_codec.txt?│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/minmax_x.idx??????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/partition.dat?????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/0_1_1_0/primary.idx???????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/1_2_2_0/checksums.txt?????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/1_2_2_0/columns.txt???????????????????│
│?default:store/b47/b47ee9c8-afa1-41ac-adfb-2e0dc42de819/1_2_2_0/count.txt?????????????????????│
└──────────────────────────────────────────────────────────────────────────────────────────────┘

10?rows?in?set.?Elapsed:?0.002?sec.?

系统表system.merge_tree_metadata_cache可用于查询rocksdb缓存的part元数据。其中

key: part元数据文件的路径，格式为<disk_name>.<relative_path>, 其中disk_name为配置在config.xml中的disk名称。如果该表使用默认存储策略，disk_name的值为default。relative_path为part元数据文件相对disk根目录的路径
value: part元数据文件的内容

测评

接下来我们分别测试在不使用和使用metadata cache的情况下，clickhouse-server的启动速度

环境准备

为了尽量模拟生产环境，在单个clickhouse实例上创建2000张MergeTree表，每张MergeTree表50个字段，350个part。单实例上一共70万个part，与笔者的生产环境持平。clickhouse版本选择22.4

脚本如下，用它分别初始化clickhouse实例A(禁用启动加速)和实例B(使用启动加速)

#!/bin/bash
?
num_tables=2000
num_columns=50
num_partitions=350

#?禁用启动加速特性
#?use_metadata_cache=0

#?使用启动加速特性
use_metadata_cache=1
function?get_create_table_query()
{
????id=$1
????echo?-n?"CREATE?TABLE?test.test_${id}?(day?Date"
????for?i?in?`seq?1?$num_columns`;?do
????????echo?-n?",?col${i}?String"
????done
????echo?-n?")?ENGINE?=?MergeTree()?PARTITION?BY?toYYYYMMDD(day)?ORDER?BY?col1"
????
????if?[?$use_metadata_cache?-eq?1?];?then?
?????echo?-n?"?SETTINGS?use_metadata_cache?=?true;"
????else
?????echo?-n?"?SETTINGS?use_metadata_cache?=?false;"
????fi
}
?
function?get_insert_table_query()
{
????id=$1
????offset=$2
????echo?-n?"insert?into?test.test_${id}?values?(today()-${offset}"
????for?i?in?`seq?1?$num_columns`;?do
????????echo?-n?",?'hello?world:${i}'"
????done
????echo?-n?");"
}
?
tock="clickhouse-client"
echo?"drop?database?if?exists?test;"?|?tock
echo?"create?database?if?not?exists?test?ENGINE?=?Ordinary;"?|?tock
?
for?i?in?`seq?1?$num_tables`;?do
????sql=`get_create_table_query?$i`
????echo?"$sql"?|?tock
?
????for?offset?in?`seq?1?$num_partitions`;?do
????????(sql=`get_insert_table_query?$i?$offset`;?echo?"$sql"?|?tock)&
????????if?((?$offset?%?90?==?0?));?then
????????????wait
????????fi
????done
????wait
done