[大数据]datax实现mysql增量同步到clickhouse

默认编辑的datax是不支持clickhouse的，
1.下载ithub上第三方的代码源
https://github.com/zhangchangwei/DataX

git clone  https://github.com/zhangchangwei/DataX.git

2、下载DataX源码：

$ cd  {DataX_source_code_home}
$ mvn -U clean package assembly:assembly -Dmaven.test.skip=true

3.可能会编译报错
去除报错编译的插件，我们只需要clickhouse相关的即可
vi pom.xml （删除报错的）
打包成功后的DataX包位于 {DataX_source_code_home}/target/datax/datax/ ，结构如下：

$ cd  {DataX_source_code_home}
$ cd  DataX/target/datax/datax/plugin/writer

编译完成会，拷贝到原先的datax插件目录下

 $  cp -r   clickhousewriter   /usr/local/datax/plugin/writer/

chown修改权限归属

这样基本就完成了，其他就在datax-web完成，采用时间戳增量，参考上几次的文章
具体json如下：

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3,
        "byte": 1048576
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "123",
            "password": "123",
            "connection": [
              {
                "jdbcUrl": [
                  "jdbc:mysql://192.168.1.1:3306/test?serverTimezone=GMT%2B8&verifyServerCertificate=false&useSSL=false"
                ],
                "querySql": [
                  "select * from rpt_supplier_statistics_data where updated_at >= FROM_UNIXTIME(${lastTime}) and updated_at < FROM_UNIXTIME(${currentTime})"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "clickhousewriter",
          "parameter": {
            "writeMode": "update",
            "username": "123",
            "password": "123",
            "column": [
              "id",
              "supplier_id",
              "main_account_id",
              "mobile",
              "`company_name`",
              "regist_time",
              "publish_pro_num",
              "open_trade",
              "month_login_num",
              "month_pv",
              "month_uv",
              "statistics_date",
              "created_at",
              "updated_at"
            ],
            "connection": [
              {
                "table": [
                  "eft_rpt_supplier_statistics_data"
                ],
                "jdbcUrl": "jdbc:clickhouse://192.168.3.102:8123/testnew"
              }
            ]
          }
        }
      }
    ]
  }
}