fix(canal):fix endless local binlog mode with exception.

add Rust canal client support (#2518 )
Revert "修复kafka消费回滚的bug (#2521 )" (#2526 )
2020-02-27 17:56:10 +08:00 · 2020-01-15 20:12:28 +08:00 · 2020-01-02 17:04:57 +08:00 · 2020-01-02 16:15:47 +08:00 · 2019-12-26 19:29:48 +08:00 · 2019-11-04 10:20:00 +08:00
338 changed files with 6448 additions and 17625 deletions
@@ -5,11 +5,13 @@
 [![average time to resolve an issue](http://isitmaintained.com/badge/resolution/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "average time to resolve an issue")
 [![percentage of issues still open](http://isitmaintained.com/badge/open/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "percentage of issues still open")

-## 背景
+## 简介

-早期，阿里巴巴 B2B 公司因为存在杭州和美国双机房部署，存在跨机房同步的业务需求 ，主要是基于trigger的方式获取增量变更。从 2010 年开始，公司开始逐步尝试数据库日志解析，获取增量变更进行同步，由此衍生出了增量订阅和消费业务，从此开启一段新纪元。
+![](https://img-blog.csdnimg.cn/20191104101735947.png)

-当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
+**canal [kə'næl]**，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费
+
+早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。

 基于日志增量订阅和消费的业务包括
 - 数据库镜像
@@ -18,12 +20,7 @@
 - 业务 cache 刷新
 - 带业务逻辑的增量数据处理

-## 项目介绍
-
-#### 名称：canal [kə'næl]
-#### 译意： 水道/管道/沟渠
-#### 产品定位： 基于数据库增量日志解析，提供增量数据订阅和消费
-#### 关键词： MySQL binlog parser / real-time / queue&topic / index build
+当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

 ## 工作原理

@@ -35,7 +32,6 @@
 - MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据

 #### canal 工作原理
-![](http://dl.iteye.com/upload/attachment/0080/3107/c87b67ba-394c-3086-9577-9db05be04c95.jpg)

 - canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议
 - MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
@@ -43,7 +39,7 @@

 ## 重要版本更新说明

-canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:
+1. canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:

 - 整体性能测试&优化,提升了150%. #726 参考: [Performance](https://github.com/alibaba/canal/wiki/Performance)
 - 原生支持prometheus监控 #765 [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
@@ -51,6 +47,8 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
 - 原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析) 参考: [Aliyun RDS QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
 - 原生支持docker镜像 #801 参考: [Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)

+2.  canal 1.1.4版本，迎来最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal动态管理能力，支持配置、任务、日志等在线白屏运维能力，具体文档：[Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
+
 ## 文档

 - [Home](https://github.com/alibaba/canal/wiki/Home)
@@ -60,6 +58,11 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
  - [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart")
  - [Aliyun RDS for MySQL QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
  - [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
+- Canal Admin
+  - [Canal Admin QuickStart](https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart)
+  - [Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
+  - [Canal Admin ServerGuide](https://github.com/alibaba/canal/wiki/Canal-Admin-ServerGuide)
+  - [Canal Admin Docker](https://github.com/alibaba/canal/wiki/Canal-Admin-Docker)
 - [AdminGuide](https://github.com/alibaba/canal/wiki/AdminGuide)
 - [ClientExample](https://github.com/alibaba/canal/wiki/ClientExample)
 - [ClientAPI](https://github.com/alibaba/canal/wiki/ClientAPI)
@@ -81,6 +84,7 @@ canal 特别设计了 client-server 模式，交互协议使用 protobuf 3.0 , c
 - canal go客户端: [https://github.com/CanalClient/canal-go](https://github.com/CanalClient/canal-go)
 - canal php客户端: [https://github.com/xingwenge/canal-php](https://github.com/xingwenge/canal-php)
 - canal Python客户端：[https://github.com/haozi3156666/canal-python](https://github.com/haozi3156666/canal-python)
+- canal Rust客户端：[https://github.com/laohanlinux/canal-rs](https://github.com/laohanlinux/canal-rs)

 canal 作为 MySQL binlog 增量获取和解析工具，可将变更记录投递到 MQ 系统中，比如 Kafka/RocketMQ，可以借助于 MQ 的多语言能力 

@@ -92,9 +96,4 @@ canal 作为 MySQL binlog 增量获取和解析工具，可将变更记录投递
 - [阿里巴巴去 Oracle 数据迁移同步工具: yugong](http://github.com/alibaba/yugong)

 ## 问题反馈
- 邮件: jianghang115@gmail.com
 - 报告 issue: [github issues](https://github.com/alibaba/canal/issues)
-
-## 其他资料
-
-  - [ ADC 阿里技术嘉年华分享 ppt ( google docs)](https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit?usp=sharing) 
@@ -1,14 +1,14 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <parent>
-    <artifactId>canal-admin</artifactId>
+    <artifactId>canal.admin</artifactId>
    <groupId>com.alibaba.otter</groupId>
-    <version>1.1.4</version>
+    <version>1.1.5-SNAPSHOT</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>

-  <artifactId>canal-admin-ui</artifactId>
-
+  <artifactId>canal.admin-ui</artifactId>
+  <name>canal admin-ui module for otter ${project.version}</name>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
bucketli	7ac605242e	fix(canal):fix endless local binlog mode with exception.	2020-02-27 17:56:10 +08:00
Rg	70d8a379f2	add Rust canal client support (#2518 )	2020-01-15 20:12:28 +08:00
rewerma	b6333f018f	Revert "修复kafka消费回滚的bug (#2521 )" (#2526 ) This reverts commit `31acfc0101`.	2020-01-02 17:04:57 +08:00
rewerma	31acfc0101	修复kafka消费回滚的bug (#2521 )	2020-01-02 16:15:47 +08:00
rewerma	6d9ea57186	修复子线程阻塞造成不能中止的问题 (#2508 )	2019-12-26 19:29:48 +08:00
Li Qiang	c4c0255073	Update README.md	2019-11-04 10:20:00 +08:00
agapple	1fa9020599	format	2019-10-15 21:38:48 +08:00
agapple	dd7554374f	format	2019-10-15 21:35:57 +08:00
shicongyang	65b5325ca6	建议增加一个属性，针对根据某一字段hash时，hash算法可以不根据database算出 #2248 (#2294 )	2019-10-15 21:32:42 +08:00
agapple	b64a3e301a	fixed issue #2286 , ArithmeticException zero	2019-10-10 15:37:17 +08:00
agapple	31562919bb	rename canal-admin project to admin	2019-10-09 14:50:08 +08:00
agapple	9c75e25da7	optimizer pom dependency	2019-10-09 14:18:37 +08:00
agapple	7d91850be6	fixed issue #2257 , time 100 parse bugfix	2019-10-09 13:39:22 +08:00
agapple	4af3b24b69	fixed issue #2281 , null reject	2019-10-08 18:50:37 +08:00
craigwu9	e4e66b143c	make canal.admin.port canal.admin.user canal.admin.passwd can be override by system env properties (#2274 ) * Update SimpleAdminConnectors.java add more information about which server execute failed * Update CanalStarter.java make canal.admin.port canal.admin.user canal.admin.passwd can be override by system env	2019-09-30 20:45:00 +08:00
agapple	dd93e2b0c4	fixed issue #2258 , optimizer rabbitmq send perf & matchTopics only match one Topic	2019-09-30 20:42:52 +08:00
agapple	d505b5ab9d	fixed issue #2258 , optimizer rocketmq send perf	2019-09-30 20:19:25 +08:00
agapple	0fb717a97f	fixed issue #2258 , optimizer kafka send perf	2019-09-30 14:24:59 +08:00
一杯Hash	c634f98fca	bugfix mysql >= 8.0, 缓存层移除，EbeanServer 运行 Fail，Unknown system variable query_cache_size error！ (#2264 ) threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLException: Unknown system variable 'query_cache_size'	2019-09-29 10:51:17 +08:00
rewerma	460479390a	bug fix (#2249 )	2019-09-26 10:58:28 +08:00
Li Qiang	26013e239d	Update README.md	2019-09-25 19:49:22 +08:00
agapple	46fe40ed47	use es6x/es7x instead of elaticsearch project	2019-09-25 19:31:52 +08:00
Koshoooooo	0c27bcf73e	Fix issue #2221 (#2242 ) fastjson修复了拒绝服务安全问题，升级到当前1.2.58的兼容安全版本1.2.58.sec06	2019-09-25 19:28:13 +08:00
agapple	d86b0e6e67	fixed testcase	2019-09-25 19:16:33 +08:00
EdgarScott	fff623acb1	Add unit tests (#2235 ) * Add Unit Tests for the following classes: com.alibaba.otter.canal.client.adapter.support.JdbcTypeUtilTest com.alibaba.otter.canal.client.adapter.support.UtilTest com.taobao.tddl.dbsync.binlog.JsonConversion_Json_ValueTest com.taobao.tddl.dbsync.binlog.JsonDiffConversionTest com.taobao.tddl.dbsync.binlog.LogBufferTest com.taobao.tddl.dbsync.binlog.LogEventTest com.taobao.tddl.dbsync.binlog.event.LogHeaderTest com.taobao.tddl.dbsync.binlog.event.RowsLogBufferTest com.alibaba.otter.canal.parse.driver.mysql.packets.client.RegisterSlaveCommandPacketTest com.alibaba.otter.canal.parse.driver.mysql.utils.ByteHelperTest com.alibaba.otter.canal.parse.inbound.mysql.ddl.DdlResultTest com.alibaba.otter.canal.parse.inbound.mysql.ddl.DruidDdlParserTest These tests were written using Diffblue Cover * Add dependencies and refactor	2019-09-25 18:53:24 +08:00
rewerma	f45c0206da	Add elasticsearch 7.x version adapter support (#2246 ) * init es data sync upgrade * Add elasticsearch 7.x version adapter support	2019-09-25 18:51:34 +08:00
gaecfov	61726c7e05	fix: ES配置文件监控问题 (#2202 ) * fix: ES监控配置文件时如果第一次因为文件错误新增失败后，再次修改不会新增的问题 ES监控配置文件添加的Config缓存没有带Instance导致匹配不到的问题 * fix：ES配置文件监控不生效	2019-09-17 09:22:55 +08:00
耐小心	79552e0a37	rabbitmq(amqp)MQ支持 (#2156 )	2019-09-16 12:43:02 +08:00
agapple	5ec2087257	fixed issue #2197 , startup.bat fix	2019-09-16 09:58:27 +08:00
rewerma	5214257cd4	ES子表含group的where条件拼接 (#2190 ) * ES子表含group的where条件拼接 * modify	2019-09-16 09:47:02 +08:00
gaecfov	365ca4d01f	fix：当查询列包含常量拼接时解析主键字段进行条件拼接时会多出 where null=null (#2188 )	2019-09-16 09:44:59 +08:00
agapple	419a73b3de	fixed issue #2168 , ignore create database	2019-09-12 09:08:30 +08:00
Jony	c093fbb8f6	动态刷新同步锁状态 (#2181 ) * 解决修改数据库远程配置，canal-adapter 一直循环destroy、init。 * 解决动态添加instance和对应的adapter无锁状态问题。	2019-09-12 08:52:33 +08:00
rewerma	31bb85bea3	fix bug #2023 and code optimize (#2167 ) * 减少等待刷新状态时间 * fix bug #2023	2019-09-12 08:47:22 +08:00
rewerma	803565456f	fix modified_time null value error (#2157 ) * template config load * compile vue frontend for product * fix modified_time null value error * modify	2019-09-07 21:53:11 +08:00
Li Qiang	520f5771d8	Update README.md	2019-09-06 15:02:34 +08:00
Li Qiang	5fbd6f6606	Update README.md	2019-09-06 14:57:47 +08:00
Li Qiang	bc9c7f0f1d	Update README.md	2019-09-06 14:54:22 +08:00
Li Qiang	08a2f0aa44	Update README.md	2019-09-06 14:45:33 +08:00
Li Qiang	a0387faa67	Update README.md	2019-09-06 14:45:14 +08:00
Li Qiang	1dd4674ccb	Update README.md	2019-09-06 13:36:04 +08:00
Jony	60e26d1ddd	解决修改数据库远程配置，canal-adapter 一直循环destroy、init。 (#2137 )	2019-09-04 12:25:54 +08:00
agapple	ecdab76705	fixed issue #2129 , Negative seek offset	2019-09-03 08:51:21 +08:00
agapple	403fdf997c	Update README.md	2019-09-02 19:28:49 +08:00
agapple	a9e83463e0	Update README.md	2019-09-02 19:27:50 +08:00
agapple	2bde4327c7	[maven-release-plugin] prepare for next development iteration	2019-09-02 14:24:16 +08:00