[大数据]etcd调研

我们这边准备将etcd作为数据库来进行存储，因此这边对etcd进行调研

简介

etcd是一款基于raft算法实现的强一致性CP系统，使用go语言编写，为分布式系统或集群访问的数据提供了可靠的方式

版本：v2 和 v3对比

在etcd进展中，在v2版本基本功能都已成熟，只是在一些情况下存在一些问题，因此就催生了v3版本

一、快速入门

我们这里不再尝试v2的功能，反正v3版本这里也都有，只是api前缀不同

1. 下载

docker pull bitnami/etcd:latest

2. 运行

docker run -d --name etcd-server \
    --publish 2379:2379 \
    --publish 2380:2380 \
    --env ALLOW_NONE_AUTHENTICATION=yes \
    --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 \
    bitnami/etcd:latest

3. 基本使用：http

http是v3里面的gRPC的网关功能，对外提供的是http方式，不过在不同的v3版本里面对应的api也是不一样的

v3.2及之前：[client-url]/v3alpha/*
v3.3：[client-url]/v3beta/，支持[client-url]/v3alpha/
v3.4：[client-url]/v3/，支持[client-url]/v3beta/，弃用[client-url]/v3beta/*
v3.5及以后：仅支持[client-url]/v3/，弃用[client-url]/v3beta/，弃用[client-url]/v3beta/*

我们这里使用v3.5部分的api，其中官网的所有api代码在这里
https://github.com/etcd-io/etcd/blob/release-3.5/api/etcdserverpb/rpc.proto，对于详细的所有api使用方面，这里不在尝试，详情见这里

1. CURD

新增

curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "a2V5MQ==", "value": "dmFsdWUx"}'

其中的值都是base64编码后的，对于其中的结构建议看上面的api链接部分

删除

curl -L http://localhost:2379/v3/kv/deleterange -X POST -d '{"key": "a2V5YQ=="}'

查询：指定的key

curl -L http://localhost:2379/v3/kv/range -X POST -d '{"key": "a2V5MQ=="}'

{
    "header":{
        "cluster_id":"14841639068965178418",
        "member_id":"10276657743932975437",
        "revision":"44",
        "raft_term":"6"
    },
    "kvs":[
        {
            "key":"a2V5MQ==",
            "create_revision":"43",
            "mod_revision":"44",
            "version":"2",
            "value":"dmFsdWUx"
        }
    ],
    "count":"1"
}

其中参数range中的结构为如下

查询：key范围

比如我们插入三条数据

curl -L http://localhost:2379/v3/kv/put  -X POST -d '{"key": "a2V5YQ==", "value": "dmFsdWVh"}'
curl -L http://localhost:2379/v3/kv/put  -X POST -d '{"key": "a2V5Yg==", "value": "dmFsdWVh"}'
curl -L http://localhost:2379/v3/kv/put  -X POST -d '{"key": "a2V5Yw==", "value": "dmFsdWVh"}'

对应的key-value为

• keya: valuea
• keyb: valueb
• keyc: valuec

搜索时候，按照范围进行搜索，其中range_end就是字符对应的ascii码+1
key: key
range_end：keyd

curl -L http://localhost:2379/v3/kv/range -X POST -d '{"key": "a2V5YQ==", "range_end": "a2V5ZA=="}'

2. watch

curl -L http://localhost:2379/v3/watch -X POST -d '{"create_request": {"key": "a2V5YQ=="}}'

$ curl -L http://localhost:22379/v3/watch -X POST -d '{"create_request": {"key": "a2V5MQ=="}}'
// 监听key的值变化，不管对应的值有没有变化都会监听到
{"result":{"header":{"cluster_id":"4396103878097145774","member_id":"13009361124239308915","revision":"8","raft_term":"2"},"events":[{"kv":{"key":"a2V5MQ==","create_revision":"2","mod_revision":"8","version":"7","value":"a2V5MQ=="}}]}}
// 监听key的删除
{"result":{"header":{"cluster_id":"4396103878097145774","member_id":"13009361124239308915","revision":"9","raft_term":"2"},"events":[{"type":"DELETE","kv":{"key":"a2V5MQ==","mod_revision":"9"}}]}}

3. key过期

// 创建租约，TTL的单位是秒
curl -L http://localhost:2379/v3/lease/grant -X POST -d '{"TTL": 25}'

{
    "header":{
        "cluster_id":"14841639068965178418",
        "member_id":"10276657743932975437",
        "revision":"62",
        "raft_term":"6"
    },
    // 租约id
    "ID":"7587859945718655050",
    "TTL":"25"
}

其中的ID就是创建好的租约，再创建kv的时候，将租约id添加进去即可

// 使用租约创建kv
curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "a2V5Yw==", "value": "dmFsdWVi", "lease": "7587859945718655050"}'

// 查询
curl -L http://localhost:2379/v3/kv/range -X POST -d '{"key": "a2V5Yw=="}'

续租

etcd这里对key的失效，采用的是租约，而租约这里可以定时的进行续租，调用如下方法即可

curl -L http://localhost:2379/v3/lease/keepalive -X POST -d '{"ID": "7587859945718655050"}'

4. 事务

这里api比较复杂，网上的例子借鉴下

# target CREATE
curl -L http://localhost:2379/v3/kv/txn \
  -X POST \
  -d '{"compare":[{"target":"CREATE","key":"Zm9v","createRevision":"2"}],"success":[{"requestPut":{"key":"Zm9v","value":"YmFy"}}]}'
# {"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"3","raft_term":"2"},"succeeded":true,"responses":[{"response_put":{"header":{"revision":"3"}}}]}

4. 授权验证：角色的访问控制

etcd对安全方面，提供三种维度

• 用户：默认root
• 角色：默认root
• 授权：

有这三个角色，但是角色和用户在初始时候还是要创建

验证初始化

对于命令的使用方面，见后面有全量etcdctl的命令使用方面

// 添加用户root
$ etcdctl add user root

// 添加角色root
$ etcdctl add role root

// 给角色root授权root角色
$ etcdctl user grant-role root root

// 可以查看用户
$ etcdctl user get root
输出：
User: root
Roles: root

验证开启

// 开启鉴权
$ etcdctl auth enable
Authentication Enabled

// 关闭鉴权
$ etcdctl auth disable
Authentication Disabled

提示：
一旦开启授权之后，后面所有的命令操作就需要带上用户 --user <user name>:<user password>

// 未开启鉴权
$ etcdctl user list

// 开启鉴权
$ etcdctl --user root:root user list

角色

etcdctl role <subcommand> [flags]

subcommand:
	add								Adds a new role
	delete						Deletes a role
	get								Gets detailed information of a role
	grant-permission	Grants a key to a role
	list							Lists all roles
	revoke-permission	Revokes a key from a role

// 添加角色
$ etcdctl role add <role name>

// 删除角色
$ etcdctl role delete <role name>

// 角色列表
$ etcdctl role list

// 给角色授权某个key，其中 <handle type> 为如下三种
// read
// write
// readwrite
$ etcdctl role grant-permission <role name> <handle type> key-name

// keys的范围是[key1, key5)
$ etcdctl role grant-permission <role name> <handle type> key1 key5

// keys的范围是[/foo/, /foo0)
$ etcdctl role grant-permission <role name> --prefix=true <handle type> /foo/

用户

$ etcdctl user <subcommand> [flags]

API VERSION:
	3.5


COMMANDS:
	add		Adds a new user
	delete		Deletes a user
	get		Gets detailed information of a user
	grant-role	Grants a role to a user
	list		Lists all users
	passwd		Changes password of user
	revoke-role	Revokes a role from a user
  
// 添加用户
$ etcdctl user add <user name>

// 删除用户
$ etcdctl user delete <user name>

// 查询用户
$ etcdctl user list

// 更新用户密码
$ etcdctl user passwd <user name>

// 给用户授权角色
$ etcdctl user grant-role <user name> <role name>

// 给用户撤销角色
$ etcdctl user grant-role <user name> <role name>

演示

1. go-client

给用户admin分配的角色admin-role，只可以操作如下范围的key，其他的都是禁止的

$ etcdctl --user root:root role get admin-role
Role admin-role
KV Read:
	[key1, key5)
KV Write:
	[key1, key5)

// 客户端使用时候添加账户即可
config = clientv3.Config{
	Endpoints:   []string{"localhost:2379"},
	DialTimeout: 5 * time.Second,
	Username: "admin",
	Password: "admin",
}

// 其中key1~key4 都可以
// key5报异常：etcdserver: permission denied
func TestEtcd1(t *testing.T) {
	res, err := etcdClient.Put(Ctx, "key5", "v1")
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(res)

	res1, err := etcdClient.Get(Ctx, "key5")
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(string(res1.Kvs[0].Value))
}

2. restful

rest方面的话，在认证方面采用的是先获取token，然后每次请求时候带上这个token

获取token

// 获取token
curl -L http://localhost:2379/v3/auth/authenticate -X POST -d '{"name": "admin", "password": "admin.123"}'

{
    "header":{
        "cluster_id":"14841639068965178418",
        "member_id":"10276657743932975437",
        "revision":"74",
        "raft_term":"6"
    },
    "token":"WQtRvsfXoaGtvrWA.298"
}

添加key1，该角色只能操作key1~key4的kv，如下为key1的操作

curl -L http://localhost:2379/v3/kv/put -X POST -H 'Authorization:WQtRvsfXoaGtvrWA.298' -d '{"key": "a2V5MQ==", "value": "dmFsdWUx"}'

5. etcdctl命令所有命令

# etcdctl help
NAME:
	etcdctl - A simple command line client for etcd3.

USAGE:
	etcdctl [flags]

VERSION:
	3.4.14

API VERSION:
	3.4

COMMANDS:
	alarm disarm						Disarms all alarms
	alarm list							Lists all alarms
	auth disable						Disables authentication
	auth enable							Enables authentication
	check datascale					Check the memory usage of holding data for different workloads on a given server endpoint.
	check perf							Check the performance of the etcd cluster
	compaction							Compacts the event history in etcd
	defrag									Defragments the storage of the etcd members with given endpoints
	del											Removes the specified key or range of keys [key, range_end)
	elect										Observes and participates in leader election
	endpoint hashkv					Prints the KV history hash for each endpoint in --endpoints
	endpoint health					Checks the healthiness of endpoints specified in `--endpoints` flag
	endpoint status					Prints out the status of endpoints specified in `--endpoints` flag
	get											Gets the key or a range of keys
	help										Help about any command
	lease grant							Creates leases
	lease keep-alive				Keeps leases alive (renew)
	lease list							List all active leases
	lease revoke						Revokes leases
	lease timetolive				Get lease information
	lock										Acquires a named lock
	make-mirror							Makes a mirror at the destination etcd cluster
	member add							Adds a member into the cluster
	member list							Lists all members in the cluster
	member promote					Promotes a non-voting member in the cluster
	member remove						Removes a member from the cluster
	member update						Updates a member in the cluster
	migrate									Migrates keys in a v2 store to a mvcc store
	move-leader							Transfers leadership to another etcd cluster member.
	put											Puts the given key into the store
	role add								Adds a new role
	role delete							Deletes a role
	role get								Gets detailed information of a role
	role grant-permission		Grants a key to a role
	role list								Lists all roles
	role revoke-permission	Revokes a key from a role
	snapshot restore				Restores an etcd member snapshot to an etcd directory
	snapshot save						Stores an etcd node backend snapshot to a given file
	snapshot status					Gets backend snapshot status of a given file
	txn											Txn processes all the requests in one transaction
	user add								Adds a new user
	user delete							Deletes a user
	user get								Gets detailed information of a user
	user grant-role					Grants a role to a user
	user list								Lists all users
	user passwd							Changes password of user
	user revoke-role				Revokes a role from a user
	version									Prints the version of etcdctl
	watch										Watches events stream on keys or prefixes
  
OPTIONS:
      --cacert=""													verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""														identify secure client using this TLS certificate file
      --command-timeout=5s								timeout for short running command (excluding dial timeout)
      --debug[=false]											enable client-side debug logging
      --dial-timeout=2s										dial timeout for client connections
  -d, --discovery-srv=""									domain name to query for SRV records describing cluster endpoints
      --discovery-srv-name=""							service name to query when using DNS discovery
      --endpoints=[127.0.0.1:2379]				gRPC endpoints
  -h, --help[=false]											help for etcdctl
      --hex[=false]												print byte strings as hex encoded strings
      --insecure-discovery[=true]					accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]	skip server certificate verification (CAUTION: this option should be enabled only for testing purposes)
      --insecure-transport[=true]					disable transport security for client connections
      --keepalive-time=2s									keepalive time for client connections
      --keepalive-timeout=6s							keepalive timeout for client connections
      --key=""														identify secure client using this TLS key file
      --password=""												password for authentication (if this option is used, --user option should not include password)
      --user=""														username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"								set the output format (fields, json, protobuf, simple, table)

二、集群

部署

version: "3.8"
networks:
  etcd:
services:
  node1:
    image: quay.io/coreos/etcd:v3.4.14
    ports:
      - "12379:2379"
    command: >
            /usr/local/bin/etcd
            --name node1
            --data-dir /var/etcd
            --listen-client-urls http://0.0.0.0:2379
            --advertise-client-urls http://0.0.0.0:2379
            --listen-peer-urls http://0.0.0.0:2380
            --initial-advertise-peer-urls http://node1:2380
            --initial-cluster node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380
            --initial-cluster-token etcd-token
            --initial-cluster-state new
            --log-level info
    networks:
      - etcd
  node2:
    image: quay.io/coreos/etcd:v3.4.14
    ports:
      - "22379:2379"
    command: >
            /usr/local/bin/etcd
            --name node2
            --data-dir /var/etcd
            --listen-client-urls http://0.0.0.0:2379
            --advertise-client-urls http://0.0.0.0:2379
            --listen-peer-urls http://0.0.0.0:2380
            --initial-advertise-peer-urls http://node2:2380
            --initial-cluster node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380
            --initial-cluster-token etcd-token
            --initial-cluster-state new
            --log-level info
    networks:
      - etcd
  node3:
    image: quay.io/coreos/etcd:v3.4.14
    ports:
      - "32379:2379"
    command: >
            /usr/local/bin/etcd
            --name node3
            --data-dir /var/etcd
            --listen-client-urls http://0.0.0.0:2379
            --advertise-client-urls http://0.0.0.0:2379
            --listen-peer-urls http://0.0.0.0:2380
            --initial-advertise-peer-urls http://node3:2380
            --initial-cluster node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380
            --initial-cluster-token etcd-token
            --initial-cluster-state new
            --log-level info
    networks:
      - etcd

zhouzhenyong@shizi-2 ~/f/l/etcd> docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                PORTS                               NAMES
a73cd9950422        quay.io/coreos/etcd:v3.4.14   "/usr/local/bin/etcd…"   3 minutes ago       Up 3 minutes          2380/tcp, 0.0.0.0:22379->2379/tcp   etcd_node2_1
3819274eec4b        quay.io/coreos/etcd:v3.4.14   "/usr/local/bin/etcd…"   3 minutes ago       Up 3 minutes          2380/tcp, 0.0.0.0:32379->2379/tcp   etcd_node3_1
8e4e8a544bb9        quay.io/coreos/etcd:v3.4.14   "/usr/local/bin/etcd…"   3 minutes ago       Up 3 minutes          2380/tcp, 0.0.0.0:12379->2379/tcp   etcd_node1_1

检查是否可用

// 查询节点是否正常运行
$ docker exec -t etcd_node1_1 etcdctl --endpoints="http://node1:2380,http://node2:2380,http://node3:2380" endpoint health
http://node3:2380 is healthy: successfully committed proposal: took = 5.6428ms
http://node1:2380 is healthy: successfully committed proposal: took = 10.1479ms
http://node2:2380 is healthy: successfully committed proposal: took = 12.1307ms

// 查询成员列表
$ docker exec -t etcd_node1_1 etcdctl --endpoints="http://node1:2380,http://node2:2380,http://node3:2380" member list
4eaf908c58b0bd74, started, node3, http://node3:2380, http://0.0.0.0:2379, false
563feabf710c6796, started, node1, http://node1:2380, http://0.0.0.0:2379, false
b48a8904150f7873, started, node2, http://node2:2380, http://0.0.0.0:2379, false

// 查询节点的角色
$ docker exec -t etcd_node1_1 etcdctl --endpoints="http://node1:2380,http://node2:2380,http://node3:2380" --write-out=table endpoint status
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://node1:2380 | 563feabf710c6796 |  3.4.14 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| http://node2:2380 | b48a8904150f7873 |  3.4.14 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| http://node3:2380 | 4eaf908c58b0bd74 |  3.4.14 |   20 kB |      true |      false |         2 |          8 |                  8 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

测试

基本功能测试

节点1添加数据

curl -L http://localhost:12379/v3/kv/put -X POST -d '{"key": "a2V5MQ==", "value": "dmFsdWUx"}'

节点2获取数据

curl -L http://localhost:22379/v3/kv/range -X POST -d '{"key": "a2V5MQ=="}'

其他功能

其他功能与单节点的功能没有什么区别，比如认证方面的，与上面一样

三、性能压测

压测工具使用的是官方工具
https://github.com/etcd-io/etcd/blob/main/tools/benchmark/README.md

$ go get go.etcd.io/etcd/v3/tools/benchmark

$ benchmark help
benchmark is a low-level benchmark tool for etcd3.

Usage:
  benchmark [command]

Available Commands:
  help            Help about any command
  lease-keepalive Benchmark lease keepalive
  mvcc            Benchmark mvcc
  put             Benchmark put
  range           Benchmark range
  stm             Benchmark STM
  txn-put         Benchmark txn-put
  watch           Benchmark watch
  watch-get       Benchmark watch with get
  watch-latency   Benchmark watch latency

$ benchmark range key1
bench with linearizable range
 0 / 10000 B                                                                                                                                                                                                                                         !   0.00%INFO: 2022/01/20 22:55:22 [core] Channel Connectivity change to CONNECTING
INFO: 2022/01/20 22:55:22 [core] Subchannel Connectivity change to READY
INFO: 2022/01/20 22:55:22 [roundrobin] roundrobinPicker: newPicker called with info: {map[0xc00039c820:{{127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}}]}
INFO: 2022/01/20 22:55:22 [core] Channel Connectivity change to READY
 10000 / 10000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 25s

Summary:
  Total:        25.2651 secs.
  Slowest:      0.0236 secs.
  Fastest:      0.0009 secs.
  Average:      0.0025 secs.
  Stddev:       0.0014 secs.
  Requests/sec: 395.8023

Response time histogram:
  0.0009 [1]    |
  0.0032 [8323] |????????????????????????????????????????
  0.0054 [1316] |??????
  0.0077 [226]  |?
  0.0100 [80]   |
  0.0123 [32]   |
  0.0145 [13]   |
  0.0168 [4]    |
  0.0191 [1]    |
  0.0213 [2]    |
  0.0236 [2]    |

Latency distribution:
  10% in 0.0015 secs.
  25% in 0.0018 secs.
  50% in 0.0022 secs.
  75% in 0.0028 secs.
  90% in 0.0038 secs.
  95% in 0.0048 secs.
  99% in 0.0086 secs.
  99.9% in 0.0145 secs.

官方数据

实测

本机硬件指标

• 处理器：2.8GHz四核 i7处理器
• 内存：16G
• 观察工具：lazydocker（懒人docker，帮你把docker stats信息快捷展示的工具）
• 压测工具：benchmark CLI，etcd的官方压测CLI命令

读命令

# Linearizable 读取请求
benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=1 --clients=1 range YOUR_KEY --consistency=l --total=10000
benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=100 --clients=1000  range YOUR_KEY --consistency=l --total=100000

# Serializable 读取请求，使用每个成员然后将数字加起来
for endpoint in {IP_1} {IP_2} {IP_3}; do
    benchmark --endpoints=$endpoint --conns=1 --clients=1 range YOUR_KEY --consistency=s --total=10000
done
for endpoint in {IP_1} {IP_2} {IP_3}; do
    benchmark --endpoints=$endpoint --conns=100 --clients=1000 range YOUR_KEY --consistency=s --total=100000
done

示例太多了，其中非线性就不再测试了，感兴趣的同学可以自己测试
?

写命令

# 假定 IP_1 是 leader, 写入请求发到 leader
benchmark --endpoints={IP_1} --conns=1 --clients=1  put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints={IP_1} --conns=1 --clients=1  put --key-size=8 --sequential-keys --total=100000 --val-size=256

benchmark --endpoints={IP_1} --conns=100 --clients=1000  put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints={IP_1} --conns=100 --clients=1000  put --key-size=8 --sequential-keys --total=100000 --val-size=256


# 写入发到所有成员
benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=1 --clients=1  put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=1 --clients=1  put --key-size=8 --sequential-keys --total=100000 --val-size=256

benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=100 --clients=1000  put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints={IP_1},{IP_2},{IP_3} --conns=100 --clients=1000  put --key-size=8 --sequential-keys --total=100000 --val-size=256

单实例：

初始资源占用

• 内存：
• cpu

集群

初始资源占用
由于每次压测，会重新启动示例，因此初始资源占用只是其中一次数据，不过大概都差不多

注意：
每次重新压测时候，也就是每次重新启动后主节点会变化，请执行如下命令重新查看主节点是哪个

$ docker exec -t etcd_node1_1 etcdctl --endpoints="http://node1:2380,http://node2:2380,http://node3:2380" --write-out=table endpoint status
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://node1:2380 | 563feabf710c6796 |  3.4.14 |   20 kB |      true |      false |         2 |          8 |                  8 |        |
| http://node2:2380 | b48a8904150f7873 |  3.4.14 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| http://node3:2380 | 4eaf908c58b0bd74 |  3.4.14 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

四、应用场景

1. 做数据库

2. 服务发现

3. 消息发布和订阅

4. 负载均衡

5. 分布式通知和协调

6. 分布式锁

参考：

https://etcd.io/
http://www.iigrowing.cn/etcd_ying_yong_diao_yan.html
https://www.cnblogs.com/xiaoyuzhou123/p/9501603.html
api官网参考
https://etcd.io/docs/v3.5/dev-guide/api_grpc_gateway/
api实体类型
https://github.com/etcd-io/etcd/blob/release-3.5/api/etcdserverpb/rpc.proto
官方api全部
https://github.com/etcd-io/etcd/blob/main/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
https://documenter.getpostman.com/view/209548/etcd/77iZMRZ#1450d673-88d8-7ece-b65a-9c45f24b22ac
https://www.secrss.com/articles/32048
压测
https://etcd.io/docs/v2.3/benchmarks/etcd-3-demo-benchmarks/