[系统运维][docker] docker-compose安装kafka集群

前言

之前有介绍docker-compose的网络模块和使用docker-compose安装zookeeper集群.([docker] docker-compose安装zookeeper集群)

在本章, 我将介绍如何使用上期的docker网络和安装好的zk集群, 创建属于自己的kafka集群.

基础依赖

• docker & docker-compose
• zookeeper单节点 Or 集群

基本思路

思路1

由于Kafka集群, 没有官方的版本. 一般的博客, 根据考究, 一般是使用如下2个大佬制作的集群.

https://hub.docker.com/r/bitnami/kafka

• 提前下载docker镜像
  docker pull bitnami/kafka:2.7.0

PS:为了以后的发展和部署稳定性. 建议设置比较稳定的版本, 而不要使用latest版本, 因为各个版本可能某些内容不一致.

Others提前准备

• 提前创建docker网络
  docker-network create zk-net

单机 - 命令行启动

在使用集群版部署之前, 我们可以参照作者给予的部署文档进行部署测试.

• Step 1: Create a network (创建docker networks)

$ docker network create app-tier --driver bridge

• Step 2: Launch the Zookeeper server instance (部署zookeeper单机版本)

$ docker run -d --name zookeeper-server \
    --network app-tier \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    bitnami/zookeeper:latest

• Step 2: Launch the Apache Kafka server instance (部署kafka单机版本)

$ docker run -d --name kafka-server \
    --network app-tier \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
    bitnami/kafka:latest

• Step 3: Launch your Apache Kafka client instance (使用kafka的命令行工具进行测试)

$ docker run -it --rm \
    --network app-tier \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
    bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka-server:9092

PS: 此步骤, 使用docker exec -it <cointainerId> /bin/bash 进入kafka容器后, 直接调用kafka-topic.sh脚本进行测试也是可以的.

单机 - (docker-compose.yml文件)

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

集群的配置文件和单机相似. 我们就先解析单机的配置文件. 上述为官方给出的单机部署文件.

networks

• networks/app-tier: 此处主要是声明docker的相关网络. 便于下方进行使用.
• driver:bridge: 使用docker networks模式的, 网桥模式.

services

• zookeeper: zookeeper单例模式部署, 及其相关配置.
• kafka: kafla单例模式部署, 及其相关配置.
• kafka/environment: 相关环境变量.
• kafka/ports: 镜像.

集群 - (docker-compose.yml文件)

version: "3"

networks:
  zk-net:
    name: zk-net

services:
  kafka01:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka01
    hostname: kafka01
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka01:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net
  kafka02:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka02
    hostname: kafka02
    ports:
      - '9093:9092'
    environment:
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka02:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net
  kafka03:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka03
    hostname: kafka03
    ports:
      - '9094:9092'
    environment:
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka03:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net    

相关实践操作

• 启动 docker-compose up -d
• 使用kafka-topic.sh链接相关kafka集群, 并使用命令进行执行: 创建topic, 启动producer, 启动consumer等操作.

Reference

[1]. (hub-docker) bitnami/kafka