[大数据]安装ES，logstash同步MySQL数据

1. 创建普通用户

### linux创建用户，elastic不能用root用户启动

-- adduser方式为脚本方式，一行搞定
#添加用户
$ adduser chenbing
#设置密码
$ passwd chenbing
# 切换用户
$ su chenbing
# 进入到用户的根目录
$ cd ~
# 进入到chenbing同级目录
$ cd ..
# 切换root用户
$ su root
# 给chenbing设置权限
$ chmod 777 -R chenbing
# 然后切换用户
$ su chenbing

-- useradd方式有点繁琐

2. 下载解压

# 选择某一目录，下载
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.1-linux-x86_64.tar.gz
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-7.16.1-linux-x86_64.tar.gz

# 解压
$ tar -zxvf elasticsearch-7.16.1-linux-x86_64.tar.gz
$ tar -zxvf logstash-7.16.1-linux-x86_64.tar.gz

3. logstash-cbb.conf配置

# 路径：logstash-7.16.1/conf/logstash-cbb.conf
input {
	stdin { }
	jdbc {
  	# type => "test_user"
    # MySQL数据库基础配置
		jdbc_connection_string => "jdbc:mysql://localhost:3306/elk_test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&useSSL=false"
		jdbc_driver_library => "/home/chenbing/cbbUtil/mysql-connector-java-8.0.22.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    jdbc_user => "root"
    jdbc_password => "root"
    # 有设置分页的情况，注意sql不要加分号；同样，导入多表数据，只需要多写几个jdbc{},在output作判断
    statement => "select * from user"
    # statement_filepath => "/home/chenbing/cbbUtil/elk_test.sql"
    # cron表达式，'* * * * *'表示每分钟执行一次，可参考学习：https://tool.lu/crontab/
    schedule => "* * * * *"
    # 指定追踪的字段，createdate是否驼峰可根据logstash控制台日志打印看到
    tracking_column => "createdate"
    # 追踪字段的类型，目前只有数字(numeric)和时间类型(timestamp)，默认是数字类型
    tracking_column_type => "timestamp"
    #记录最后一次运行的结果
    record_last_run => true
    #上面运行结果的保存位置
    last_run_metadata_path => "/home/chenbing/cbbSoft/logstash-7.16.1/jdbc-result.txt"
	}
}
#filter {
#		json {
#				source => "message"
#       remove_field => ["message"]
#   }
#		mutate {
#    		rename => { "CreateDate" => "createdate" }
#    }
#}
output {
  elasticsearch {
    # ES的IP地址及端口
  	hosts => ["localhost:9200"]
    # 索引名称，可自定义，对应database
    index => "user"
    # 需要关联的数据库中有一个id字段(table的主键)，对应类型中的id
    document_id => "%{userid}"
    # 文档类型对应table
    document_type => "_doc"
  }
  stdout {
    # 控制台以json格式打印
  	codec => json_lines
    # 下面这种打印格式会看起来更舒服些
    # codec => rubydebug
  }
}

4. 启动

### 1、启动elastic
# 进入es解压包，执行如下启动ES
# 直接当前终端启动,进入elasticsearch-7.16.1目录
$ /bin/elasticsearch
# 如要将 Elasticsearch 作为守护进程运行，请-d在命令行中指定，并使用以下-p选项将进程 ID 记录在文件中，官网方式
# 启动可能会报错，控制台日志显示是需要配置进入config/elasticsearch.yml配置，配置内容见附录
./bin/elasticsearch -d -p pid




### 2、启动logstash,进入logstash-7.16.1目录
$ bin/logstash -f config/logstash-cbb.conf 

5. 关闭

### 1、两种方式关闭elastic

#### 方式一
# 查看java进程，elasticsearch前面的数字就是进程号
$ jps
$ kill -9 进程号

#### 方式二
# 第一条记录为elastic进程信息,第一列为elastic的用户，第二列为进程号
$ ps -ef | grep elastic
$ kill -9 进程号


### 2、关闭logstash，同样也可以试试上面的方法

附录

config/elasticsearch.yml配置

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#bootstrap.memory_lock: true
#ES_JAVA_OPTS: "-Xms512m -Xmx1024m"
#MAX_LOCKED_MEMORY: unlimited
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts: ["0.0.0.0", "[::1]"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html

运行报错vm.max_map_count [65530] is too low

# 在/etc/sysctl.conf文件最后添加一行:vm.max_map_count=262144
$ vim /etc/sysctl.conf
# 立即生效
$ /sbin/sysctl -p

办法千万种，解决最重要。