kafka 2.8.0 without zookerper 部署
1.前言
Kafka 2.8.0 可以不需要zookerper，没有zk的参与将会使kafka的吞吐量得到更大的提高，具体可看官网，这篇文章主要是针对kafka 2.8.0 kafka KRaft mode 的安装部署，以三台机器为例，由于网上资料较少，大部分都是靠自己摸索，若文中有理解不对的地方，可随时赐教，共同探讨。
2.下载安装包 (https://kafka.apache.org/downloads)
3.解压安装包
$ tar -zxvf kafka_2.12-2.8.0.tgz -C /data1/opt/module
4.修改server.properties文件
$ cd /data1/opt/module/kafka_2.12-2.8.0/config/kraft
$ vim server.properties
注意：
- 设置node.id,集群中node.id不可重复
- 设置controller.quorum.voters，所有controller需要全部配置
- log.dirs可配置多目录
- 配置文件中所有localhost,更改为ip或主机名
5.创建cluster ID
在任意一台机器执行
$ ./bin/kafka-storage.sh random-uuid
6.同步cluster ID
在所有节点执行
$ ./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties
7.启动kafka
在每台节点上执行，启动kafka
$ ./bin/kafka-server-start.sh -daemon /data1/opt/module/kafka_2.12-2.8.0/config/kraft/server.properties
8.指令测试
(1) 查看当前服务器中的所有topic
$ ./bin/kafka-topics.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --list
__consumer_offsets
four
one
three
two
(2) 创建topic
$ ./bin/kafka-topics.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --create --replication-factor 2 --partitions 2 --topic five
Created topic five.
(3) 删除topic
$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --delete --topic one
(4) 发送消息
$ ./bin/kafka-console-producer.sh --broker-list hadoop101,hadoop102,hadoop103 --topic three
>one
>two
>three
>
(5) 消费消息
$ ./bin/kafka-console-consumer.sh --bootstrap-server hadoop101,hadoop102,hadoop103 --topic three –group g1
$ ./bin/kafka-console-consumer.sh --bootstrap-server hadoop101 --from-beginning --topic three
one
three
saas
two
five
(6) 查看某个Topic的详情
$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --describe --topic three
Topic: three TopicId: 6enhWA-nRh6bMPmk6jWw0w PartitionCount: 2 ReplicationFactor: 2 Configs: segment.bytes=1073741824
Topic: three Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: three Partition: 1 Leader: 3 Replicas: 3,1 Isr: 3,1
(7) 修改分区数(KRaft mode 不支持)
$ ./bin/kafka-topics.sh --bootstrap-server hadoop101 --alter --topic two --partitions 2
Error while executing topic command : The broker does not support CREATE_PARTITIONS
[2021-07-20 15:07:36,742] ERROR org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support CREATE_PARTITIONS
(kafka.admin.TopicCommand$)
9.压测
(1) Kafka Producer压力测试
$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=hadoop101,hadoop102,hadoop103
100000 records sent, 49455.984174 records/sec (4.72 MB/sec), 561.53 ms avg latency, 878.00 ms max latency, 595 ms 50th, 839 ms 95th, 871 ms 99th, 877 ms 99.9th.
(2) Kafka Consumer压力测试
$ ./bin/kafka-consumer-perf-test.sh --broker-list hadoop101,hadoop102,hadoop103 --topic test --fetch-size 10000 --messages 10000000 --threads 1
2021-07-20 15:20:42:518, 2021-07-20 15:20:57:845, 9.5367, 0.6222, 100000, 6524.4340, 3556, 11771, 0.8102, 8495.4549
|